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The  purpose  of  this  manual  is  to  establish  policies  for  the 
conduct  of  Joint  Tests  and  Evaluations  and  provide  guidance 
for  the  participating  Services,  DoD  Agencies  and  the  Joint 
Test  Director  who  is  responsible  for  executing  the  test. 

The  manual  documents  systems  by  which  the  Director  Defense 
lest  and  Evaluation  (DDTE)  administers  the  JT&E  program, 
describes  the  roles  and  responsibilities  of  OSD,  the  Ser¬ 
vices  and  the  Defense  Agencies,  and  outlines  the  mission, 
organization  and  function  of  the  Joint  Test  Director  and 
the  Joint  Test  Force.  The  DDTE  procedures  and  mechanisms 
for  planning,  budgeting,  executing,  and  controlling  the 
total  JT&E  program  are  also  described  in  the  manual,  which 
is  designed  primarily  to  assist  the  Joint  Test  Director  in 
organizing  a  Joint  Test  Force  and  planning  and  executing  a 
JT&E  to  achieve  the  test  objectives. 
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CHAPTER  I 
ARCHITECTURE 


A.  INTRODUCTION 

1.  Purpose 

This  manual  is  designed  to  serve  two  major  purposes.  The  first 
is  to  document  the  system  by  which  the  Office  of  the  Secretary  of  Defense/ 
Director  Defense  Test  and  Evaluation  (OSD/DDTE)  administers  the  Joint  Test 
and  Evaluation  (JT&E)  program.  The  manual  describes  the  roles  and  respon¬ 
sibilities  of  OSD,  the  Services  and  the  Defense  Agencies.  The  second 
purpose  is  to  provide  guidance  to  a  newly-selected  Joint  Test  Director 
(J1D)  in  the  organization  and  implementation  of  a  JT&E. 

2.  Scope  of  Joint  Test  and  Evaluation 

JT&Es  are  conducted  when  required  and  initiated  by  the  DOTE.  The 
primary  purpose  of  JT&E  is  to  examine  the  capability  of  developmental  and 
deployed  systems  to  perform  their  intended  missions  in  a  joint  environment. 
JT&Es  may  also  be  conducted  to  provide  information  in  the  following  areas: 
technical  concepts  evaluation;  system  requirements;  system  improvements; 
systems  interoperability;  force  structure  planning;  testing  methodologies; 
and  doctrine,  tactics,  and  operational  procedures  for  joint  operations. 
Responsibility  for  managing  the  organization  and  implementation  of  each 
JT&E  is  delegated  to  a  specific  DoD  component  (the  Lead  Service)  and  sup¬ 
ported  by  forces  and  materiel  from  each  participating  component.  Conduct 
of  JT&E  is  in  accordance  with  established  joint  doctrine  find  consistent 
with  the  assigned  mission  of  each  participating  Service.  Tests  involving 
alternative  concepts,  organization,  tactics,  or  procedures  are  coordinated 
with  the  Office  of  the  Joint  Chiefs  of  Staff  (OJCS)  and  the  Services. 

JT&Es  must  be  structured  to  provide  valid  evaluation  data  and 
adequate  scope  for  test  and  evaluation  of  large,  complex  supporting  sys¬ 
tems.  When  possible,  the  test  environment  should  include  representative 
forces,  threat,  terrain,  climate  and  weather.  Data  acquired  from  JT&Es  are 
consolidated  with  other  information  and  used  as  a  basis  for  future  systems 
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acquisition,  mission  area  analysis,  program  decisions  on  related  forces, 
force  issues,  operational  procedures,  and  doctrine. 

The  DOTE  has  established  a  JT&E  planning,  budget,  and  review 
process  and  publishes  a  Five-Year  Test  Plan  (FYTP)  to  permit  proper  pro¬ 
gramming  and  budgeting.  The  plan  includes  the  establishment  of  priorities 
for  accomplishing  JT&E.  The  OJCS  and  the  Services  participate  in  the 
review  and  selection  process  and  assist  in  the  development  of  the  FYTP  and 
the  test  priority  list. 

The  criteria  for  JT&E  selection  are: 

•  The  feasibility  of  the  JT&E  can  be  verified  by  study. 

•  Two  or  more  Services  will  participate  in  a  joint  environment. 

0  An  appropriate  range/maneuver  area  with  a  minimum  number  of 
constraints  is  available. 

0  Forces  are  representative,  and  resources  are  available. 

0  A  desired  degree  of  realism  can  be  achieved. 

•  Funds— Operation  and  Maintenance  (O&M),  Research  and  Development 
(R&D),  and  other  funds— have  been  identified  and  programmed. 

B.  THE  POP  JOINT  TEST  AND  EVALUATION  PROGRAM 


1.  Background 

As  a  distinct  DoO  activity.  Joint  Test  and  Evaluation  originated 
in  the  1970  report  of  the  Blue  Ribbon  Defense  Panel.  The  report  recom¬ 
mended  that  continuing  responsibility  be  vested  in  an  OSD  staff  element 
dedicated  to  test  and  evaluation.  This  group  was  to  direct  realistic  test 
and  evaluation  and  to  develop  means  to  assure  that  productive  joint  testing 
would  be  accomplished.  That  recommendation  resulted  in  what  is  now  the 
Office  of  the  Director,  Defense  Test  and  Evaluation  (ODDTE).  In  1979,  DOTE 
undertook  the  structuring  and  codifying  of  a  process  which  until  that  time 
had  been  largely  ad  hoc.  The  architecture  of  the  JT&E  process  is  des¬ 
cribed  in  the  following  sections.  Two  major  aspects  of  JT&E  are  considered 
throughout.  The  cycle  of  Initiation,  execution,  and  completion  of  specific 
joint  tests  is  described.  Equally  important,  the  process  by  which  the 
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total  JT&E  program  is  integrated  into  the  Defense  program  is  also 
discussed. 

2.  Nomination.  Selection  and  Budgeting  Process 

The  ODTE  is  responsible  for  planning,  programming,  and  budgeting 
for  the  JT&E  program.  The  Services  and  Defense  Agencies  have  a  responsi¬ 
bility  to  program  and  budget  for  specific  joint  tests  which  will  involve 
their  forces.  In  order  to  provide  the  information  required  on  a  timely 
basis,  the  DOTE  has  established  a  nomination  and  selection  cycle  which  is 
aligned  with  key  action  dates  in  the  DoD  Planning,  Programming,  and  Budget¬ 
ing  System  (PPBS).  A  diagram  of  the  JT&E  cycle  is  shown  in  Figure  1-1. 
a.  Nomination  and  Selection 

The  formal  cycle  begins  each  year  when  the  DOTE  issues  a 
reminder  to  the  Services,  Defense  Agencies,  the  OJCS  and  the  OSD  staff  to 
prepare  JT&E  nominations.  By  May  1,  the  Services  submit  to  the  DOTE  the 
nominations  of  joint  tests  they  believe  should  be  undertaken  in  the  next 
five  fiscal  years.  Nominations  of  JT&Es  which  concern  issues  related  to 
joint  doctrine  or  mission  must  be  submitted  through  the  OJCS  for  coordina¬ 
tion  and  concurrence.  OSD  staff  and  Defense  Agency  nominations  are  sub¬ 
mitted  directly  to  the  OOTE  unless  they  also  concern  mission  and  doctrine 
and  must  be  forwarded  through  the  OJCS.  The  CINCs  of  the  Unified  and 
Specified  Commands  submit  all  nominations  through  the  OJCS.  Figure  1-2 
provides  a  graphic  summary  of  the  nomination  process. 

Nominations  should  be  for  the  succeeding  five  years  for 
inclusion  in  the  FYTP.  The  goal  is  a  definitive  five-year  JT&E  program 
with  new  test  start  dates,  tailored  to  test  requirements  and  budget  con¬ 
straints.  The  FYTP  will  be  updated  each  year  to  accommodate  changes 
induced  by  current  test  performance,  new  test  selections,  feasibility  study 
results,  and  the  funding  implications  of  each.  Nominations  should  provide 
the  information  contained  in  the  sample  format  in  Figure  1-3. 

(1)  Planning  Committee 

After  receiving  the  nominations,  the  DOTE  convenes  the 
Planning  Committee  to  review  the  proposed  nominations.  The  membership  and 
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Figure  1-1.  Joint  Test  and  Evaluation  Planning  Cycle 


JT8.E 

NOMINATIONS 


I.  TTTLE 


II.  TEST  08JECT1VE— ISSUES  TO  BE  ADDRESSED— UTILITY  &  IMPACT  IF  OBJECTIVES  ARE  MET 


III.  RECOMMENOED  TEST  DATE(S)  <INCLUSIVE)/MILESTONES 


IV.  SPECIAL  REQUIREMENTS  (TERRAIN,  WEATHER.  CLIMATE.  ETC.) 


V.  SUGGESTED  PARTICIPANTS  (BY  SERVICE  OR  CINC) 


VI.  OUTLINE  TEST  CONCEPT 


VII.  ESTIMATE  OF  FUNOS  AND  RESOURCES 


Figure  1-3.  Sample  JTAE  Nomination  Format 
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responsibilities  of  the  Planning  Committee  are  listed  in  Figure  1-4.  The 
permanent  committee  members  will  be  from  the  Services;  nonpermanent  members 
will  be  from  Agencies  submitting  JT&E  nominations.  The  permanent  Service 
and  Agency  members  must  have  authority  to  speak  for  their  organizations  and 
will  coordinate/interface  with  their  respective  Services  and/or  Agencies. 
The  Services  and  Agencies  may  send  additional  personnel  for  information 
and/or  to  assist  the  permanent  members.  Meetings  of  the  Planning  Committee 
will  be  held  annually  or  at  the  call  of  the  Chairman.  The  Chairman  and  the 
committee  members  will  be  the  action  staff  for  the  Senior  Advisory  Council. 

(2)  Technical  Advisory  Board 

A  Defense  Technical  Advisory  Board  (TAB)  is  available 
to  advise  the  DOTE  on  technical  issues  during  the  nomination,  selection, 
conduct,  and  evaluation  of  JT&E  programs.  The  membership  and  responsibili¬ 
ties  of  the  TAB  are  listed  in  Figure  1-5.  The  TAB  consists  of  twelve 
high-level  civilian  scientists  from  the  Army,  Air  Force,  and  Navy /Marine 
Corps  who  have  current  knowledge,  experience,  and/or  responsibility  for 
advising  their  own  Service  in  the  design,  planning,  conduct,  and  evaluation 
of  large-scale  tests  and  exercises  and  who  can  provide  both  memory  and 
expertise  on  the  technical  problems  and  solutions  associated  with  such 
testing.  The  TAB  provides  recommendations  for  personnel  to  staff  smaller 
Technical  Advisory  Groups  (TAGs),  which  are  formed  to  provide  technical 
advice  to  the  Joint  Test  Director  of  each  JT&E  program.  The  TAB  supports 
the  DOTE  in  a  strictly  advisory  capacity.  It  meets  annually,  after  the 
Planning  Committee  has  submitted  its  nominations,  or  at  the  call  of  the 
DOTE. 

(3)  Senior  Advisory  Council 

Upon  completion  of  the  Planning  Committee  review,  the 
proposed  update  of  the  FYTP  (new  starts,  changes,  etc.)  will  be  presented 
to  the  Senior  Advisory  Council  (SAC)  by  the  Chairman  of  the  Planning  Com¬ 
mittee.  Membership  and  responsibilities  of  the  SAC  are  listed  in 
Figure  1-6.  The  SAC  will  provide  the  DOTE  with  recommendations  concerning 
approval /disapproval  of  the  proposed  FYTP,  Service/Agency  views,  and  speci¬ 
fication  of  the  impacts  on  the  Services  of  the  proposed  tests.  With  the 
information  provided  by  the  Senior  Advisory  Council,  the  DOTE  will  amend 
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PLANNING  COMMITTEE 


Permanent  Members:  Chairman:  ODDTE.  Deputy  0 tractor  Tactical  Air  and  Land  Warfare  Systems 

Ona  rapraaantativa  of  JCS  Staff 
Ona  rapraaantativa  of  each  Service 


Nonperm aoent  Members:  Ona  rapraaantativa  from  each  agency  submitting  nominations 

for  consideration. 


Baaponaibiiitiaa: 

1 .  Receive  JTSE  nominations  from  OOOTE. 

2.  Review  nominations  to: 

a.  Initially  determine  feasibility. 

b-  Identify  duplication  end  possible  consolidation. 

c.  Determine  adequacy  of  data  and  details  of  nominated  JT&E. 

3.  Place  nominated  tests  in  priority  order  for  presentation  to  Senior  Advisory  Council. 

4.  Review,  refine,  and  validate  ob) actives,  data  and  details  of  nominated  tests  and 
endorse  desirability  and  feasibility  of  nominated  testa. 

5.  Recommend  to  the  Senior  Advisory  Council  JT&Es  to  be  conducted  by  priority. 

S.  Develop  Five-Year  Test  Plan  and  submit  the  FYTP  to  the  Senior  Advisory  Council  for 
spproval. 

7.  Act  as  action  staff  for  the  Senior  Advisory  Council. 

8.  Coordinate: 

a.  Preliminary  test  concepts  with  nominating  Service/ Agency. 

b.  Schedules  of  nominated  tests. 

c.  Development  of  test  definition  and  test  design  with  nominating  Service/  Agency 
and  DOTE  Support  Agent,  when  designated. 

9.  Chairman.  Planning  Committee  will  be  executor  and  nonvoting  member  of  the 
Senior  Advisory  Council. 

2006/8 OW 


Figure  1-4.  Membership  and  Responsibilities  of  the  Planning  Committee 
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TECHNICAL  ADVISORY  BOARD 


Permanent  Members:  Twelve  civilian  scientists  (four  each  from  the  Army, 

Air  Force,  and  Navy/ Marine  Corps) 

Chairman:  Rotated  periodically  among  permanent 
members 


Responsibilities: 

1 .  Provide  technical  recommendations  to  the  DDTE  concerning: 

a.  JT&E  Nominations 

b.  Evaluation  Plan 

c.  Test  Design 

d.  Simulation  Plan 

e.  Instrumentation  Plan 

f.  Analysis  Plan 

2.  Recommend  personnel  to  staff  the  smaller  Technical  Advisory  Groups  (TAGs) 
created  for  each  JT&E. 

3.  Provide  technical  advice  to  the  DOTE  on  other  matters  as  required. 

4.  The  Chairman  of  the  Technical  Advisory  Board  will 
serve  as  a  technical  advisor  and  nonvoting  member  of  the 
Senior  Advisory  Council. 


Figure  1-5.  Membership  and  Responsibilities  of  the  Technical  Advisory  Board 
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SENIOR  ADVISORY  COUNCIL 


Permanent  Members:  Chairman:  Director  Defense  Test  and  Evaluation. 

One  mern5er  from  the  JCS  Staff. 

One  member  from  each  Service. 

Executive/ Advisor:  Chairman  of  the  Planning  Committee. 

Nonpermanent  Members:  One  representative  from  each  agency,  as  applicable. 


Responsibilities: 


1 .  Recommend  JT&Es  to  be  conducted  as  part  of  DoD  JT&E  program  presented  by  the 
DoD  JT&E  Planning  Committee. 

2.  Establish  priorities  and  schedules  of  JT&Es  to  be  conducted. 

3.  For  each  selected  JT&E  review: 

a.  Preliminary  test  concepts  and  objectives. 

b.  Feasibility  of  tests. 

c.  Impacts  of  resource  requirements. 

d.  Planned  budget  level  of  OSD /DOTE  funds. 

e.  Selection  of  Lead  Services  and  participating  Services. 

4.  Recommend  next  FY  starts  and  JT&E  Five-Year  Plan. 

5.  Review  and  recommend  JT&E  management  and  policy  changes. 

6.  Review  JTD  and  Independent  Evaluator's  Final  Test  Report. 


Figure  1-6. 


Membership  and  Responsibilities  of  the  Senior 
Advisory  Council 
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and  approve  the  FYTP  and  direct  the  conduct  of  feasibility  studies  on  new 
start  tests.  The  Chairman,  Senior  Advisory  Council,  will  approve  those 
JT&E  to  be  conducted  under  the  DoD  JT&E  Program.  Permanent  OJCS  and  Ser¬ 
vice  members  will  coordinate  for  their  respective  Services/Agencies. 
Nonpermanent  members  will  be  advisory  on  JT&Es  nominated  by  their  Agencies. 
Meetings  will  be  held  annually  or  at  the  call  of  the  Chairman.  The  deci¬ 
sions  of  the  DDTE  will  be  communicated  to  the  Services,  OJCS,  Defense 
Agencies,  and  the  OSD  staff. 

b.  Feasibility  Determination 

A  study  will  be  conducted  to  determine  the  feasibility  of 
accomplishing  a  JT&E  selected  by  the  DDTE.  Normally,  the  feasibility  study 
will  be  conducted  by  a  DDTE  Support  Agent.  Participating  Services  and  the 
JTD  (when  appointed)  will  work  with  the  Support  Agent  to  provide  resource 
estimates  and  advice  on  operational  matters. Occasionally,  a  test  may  be  so 
constrained  or  so  conceptually  simple,  that  it  is  judged,  prima  facie,  to 
be  feasible.  Because  of  the  potential  cost  growth  and  other  difficulties, 
such  a  judgment  is  expected  to  be  rare.  DDTE  Support  Agents  are  the  organi¬ 
zations  chosen  by  DDTE,  as  required,  to  carry  out  feasibility  studies,  test 
designs,  and  independent  evaluations.  A  support  agent  may  also  be  selected 
to  perform  an  independent  evaluation  of  the  test.  The  support  agent  may  be 
a  DoD  organization,  a  Federal  Contract  Research  Center  (FCRC),  or  a  quali¬ 
fied  contractor. 

Results  of  the  feasibility  studies  are  provided  to  the  DDTE, 
and  the  DDTE  makes  a  final  decision  concerning  test  conduct.  If  the  tests 
are  judged  feasible,  the  refined  resource  estimate  from  the  feasibility 
study  will  be  used  to  update  the  JT&E  budget  estimate.  Timing  of  feasibil¬ 
ity  studies  must  be  in  consonance  with  the  budget  cycle.  With  the  cycle 
shown  earlier  in  Figure  1-1,  the  feasibility  studies  will  terminate  in  time 
for  the  OSD  budget  submission  in  December  for  the  budget  year  and  the 
authorization  year.  Equally  important,  the  feasibility  studies  will  pro¬ 
vide  better  resource  estimates  and  the  Services  can  update  their  Program 
Objective  Memorandum  (POM)  inputs  to  reflect  a  best  estimate  of  the  JT&E 
requirements. 
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For  a  particular  joint  test,  the  end  of  the  feasibility 
determination  is  a  crucial  milestone.  If  feasible,  the  joint  test  becomes 
a  firm  entry  in  the  FYTP  and  implementing  actions  begin, 
c.  Budgeting 

Each  year,  the  DOTE  prepares  a  dynamic  working  plan  called 
the  Five-Year  Test  Plan  (FYTP).  The  FYTP  includes  the  overall  JT&E  program 
budget  with  ongoing  tests,  previously  selected  tests,  and  new  starts  organ¬ 
ized  by  fiscal  year.  Adjustments  to  the  overall  program  budget  are  made  to 
meet  fiscal  limits  established  by  the  OSD  Comptroller.  The  DDTE  tasks 
existing  Joint  Test  Forces  (JTFs)  for  budget  updates.  These  are  usually 
submitted  in  writing,  then  subsequently  briefed  by  the  JTD.  Thus,  the  DDTE 
remains  cognizant  of  budget  changes  which  have  occurred  since  the  previous 
submission  and  of  the  ramifications  of  adjusting  the  JTF's  budget,  should 
this  become  necessary. 

The  completed  FYTP  becomes  one  element  of  the  overall  DoD 
Research  Development  Test  and  Evaluation  (RDT&E)  program  which,  with  all 
other  programs,  is  subjected  to  intensive  review  within  OSD  and  the  Office 
of  Management  and  Budget  (0MB)  prior  to  becoming  a  part  of  the  President's 
Budget.  During  this  review  cycle,  adjustments  may  be  and  usually  are  made 
to  the  RDT&E  program.  In  turn,  adjustments  to  the  FYTP  may  be  required. 
If  so,  the  DDTE  readjusts  the  FYTP  and  FY  budgets  accordingly,  with  due 
consideration  given  to  the  effect  of  reduction  on  current  tests.  The  DDTE 
works  closely  with  the  JTDs  during  this  time  to  avoid  adversely  affecting 
test  plans  and  schedules.  Priority  is  normally  given  to  those  tests  in  or 
entering  the  execution  phase,  and  to  long-lead  items  which,  if  delayed, 
would  significantly  increase  the  total  cost  of  a  given  test. 

During  the  remainder  of  the  PPBS  cycle,  from  January  to 
September,  Congress  reviews  the  budget,  takes  testimony,  and  initiates  and 
enacts  authorization  and  appropriation  bills  for  presidential  signature. 
The  authority  to  fund  approved  programs  is  granted  by  0MB  through  alloca¬ 
tion  of  appropriations  to  the  federal  departments.  With  this  authority, 
offices  within  OoO  such  as  ODDTE  may  allot  funds  to  individual  JTFs  for 
purchasing  goods  and  services. 
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Service  support  to  JT&E  is  usually  drawn  from  RDT&E  and  O&M 
funds,  and  the  personnel  and  equipment  resources  are  tightly  constrained. 
In  the  past,  support  of  a  JT&E  usually  meant  that  some  training,  testing, 
or  exercise  activity  was  excluded  to  provide  the  resources  for  JT&E.  The 
JT&E  planning  process  provides  the  Services  with  knowledge  of  the  FYTP, 
thereby  allowing  the  Services  time  for  advance  planning  and  to  internally 
reallocate  resources  in  order  to  achieve  the  commitments  implied  by  the 
FYTP.  Any  RDT&E  or  procurement  Service  expenditures  required  to  support  a 
JT&E  are  subject  to  the  same  lead  time  and  review  processes  as  the  DoO  JT&E 
funds. 

3.  Implementation 

The  implementation  of  a  JT&E  may  be  separated  into  three  distinct 
processes.  These  are  test  planning,  test  execution,  and  test  reporting. 

a.  Test  Planning 

Upon  completion  of  the  feasibility  study  and  entry  into  the 
budget,  a  JT&E  enters  the  test  planning  phase.  Several  key  events  occur  at 
this  time.  The  DDTE  appoints  the  Lead  Service  and  requests  that  a  JTD  be 
nominated.  The  JTD  is  then  appointed  by  the  DDTE;  Deputy  Test  Directors 
(DTDs)  are  appointed  by  their  respective  participating  Services  in  coordi¬ 
nation  with  the  DOTE.  While  the  JTD/DTD  selection  process  is  underway,  the 
DDTE  will  prepare  a  charter  for  the  JTD  and  the  Joint  Test  Force  (JTF)  in 
coordination  with  the  appropriate  Services.  The  DDTE  will  prepare  tasking 
to  the  DDTE  Support  Agent  for  development  of  the  test  design.  This  tasking 
will  be  based  on  the  details  contained  in  the  nomination  and  on  the  feasi¬ 
bility  study.  When  the  JTD  has  been  nominated  and  approved,  the  charter 
will  be  provided  to  all  Services  involved.  Actual  preparation  of  the  test 
design  should  begin  at  this  point.  During  preparation  of  the  test  design, 
the  JTD  will  work  closely  with  the  test  designer  and  the  participating 
Services.  The  test  design  will  be  coordinated  with  the  JTD  and  the  partici 
pating  Services  to  provide  a  precise  response  to  the  DDTE  test  objectives. 
Everyone  should  be  striving  for  the  goal  of  a  test  design  that  will  achieve 
the  test  objectives  and  be  operationally  feasible  and  within  budget/ 
resource  constraints. 
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During  preparation  of  the  test  design,  the  JTD  must  also  be 
planning  and  organizing  his  staff,  deciding  upon  a  headquarters  location 
for  the  JTF,  and  selecting  appropriate  test  site(s).  By  the  time  the  test 
design  has  been  coordinated  and  approved,  the  key  members  of  the  JTD  staff 
should  be  in  place.  As  soon  as  the  personnel  resources  are  available,  the 
JTD  should  commence  development  of  the  test  plans  necessary  to  successfully 
and  reliably  conduct  the  JT&E.  The  Field  Test  Plan  contains  a  step-by-step 
description  of  the  activities  to  be  undertaken  during  the  JT&E.  An  Instru¬ 
mentation  Plan,  a  Data  Management  Plan,  a  Simulation  Plan  (if  required), 
and  a  Logistics  Support  Plan  must  also  be  prepared.  If  the  DDTE  directs  an 
independent  evaluation,  a  DOTE  Support  Agent  will  be  tasked  to  develop  an 
Independent  Evaluation  Plan,  based  on  the  approved  test  design.  The  Field 
Test  Plan  will  be  coordinated  with  the  participating  Services  and  approved 
by  the  DDTE. 

b.  Test  Execution 

Based  on  the  approved  Field  Test  Plan,  the  JTD  will  under¬ 
take  all  activities  necessary  to  meet  the  test  objectives  (see  Chapter  III, 
JT&E  Implementation).  8riefly,  the  JTD  must  prepare  and  checkout  his 
instrumentation,  data  management,  data  validation,  simulation,  and  analysis 
systems.  As  the  preparation  proceeds,  detailed  subplans  should  be  prepared 
as  necessary  and  tasking  (as  previously  coordinated)  provided  to  the  Ser¬ 
vices  for  participating  forces  and  equipment,  and  required  personnel, 
facilities,  and  administrative  support.  Finally,  test  record  trials  must 
be  conducted,  and  data  collected  «md  validated  in  accordance  with  the  Field 
Test  Plan.  Ouring  the  execution  phase,  the  JTD  must  keep  all  principals 
fully  informed,  especially  regarding  resource  taskings.  The  DDTE  will 
monitor  and  evaluate  the  test  execution. 

c.  Test  Reporting 

Throughout  the  test  execution,  the  JTD  will  submit  test 
reports  and  interim  and  special  reports,  as  required.  (See  Chapter  III. E , 
Post-Test  Activities.)  These  reports  will  document  test  conditions,  pro¬ 
gress  achieved,  and  any  expected  deficiencies  and  problems.  Upon  comple¬ 
tion  of  each  test,  reports  documenting  test  conditions  and  results  will  be 
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generated  and  submitted  to  the  DOTE  for  inclusion  in  a  JT&E  information 
file.  Two  reports,  a  JTF  Final  Test  Report  and  a  JTD  Management  Report, 
will  be  prepared. 

A  "JTF  Final  Test  Report"  will  document  the  factual  aspects 
of  the  test  such  as  environmental  test  conditions,  test  limitations,  scena¬ 
rios  played,  data  collected,  statistical  summaries,  preliminary  empirical 
results,  the  results  of  quick  look  analysis,  test  participants,  and  any 
related  items  or  comments  relevant  to  the  analysis  and  assessment  of  the 
test  results.  Evaluating  and  reporting  of  test  results  is  the  primary 
responsibility  of  the  JTD. 

A  "Management  Report"  will  document  the  JTD's  assessment  of 
the  test  objectives,  the  lessons  learned,  recommendations,  and  the  poten¬ 
tial  utility  of  the  test.  If  required,  the  DDTE  will  provide  the  results 
of  the  test  to  the  DDTE  Support  Agent  selected  to  conduct  an  independent 
evaluation.  Test  reports  will  be  provided  to  the  participating  Services 
and  the  Senior  Advisory  Council. 

4.  Management  Evaluation 

The  life  cycle  of  a  typical  JT&E  ends  in  management  evaluation 
which  includes  provision  for  the  retention  of  valuable  test  findings  and 
for  improving  the  JT&E  process.  The  management  evaluation  process  should 
provide  a  sound  basis  for  the  screening,  selection,  and  administration  of 
future  JT&Es. 

a.  Information  Control  Process 

The  DOTE  will  establish  and  maintain  a  JT&E  information  file 


containing  a  library  of  completed  test  and  evaluation  reports  and  identify¬ 
ing  the  data  bases  developed  and  computer  models  created.  Through  the 
library,  the  DOTE  will  be  the  focal  point  for  JT&E  data  and  will  provide 
information  exchange  service  to  interested  parties.  The  DDTE  will  prepare 
an  Annual  Report  and  submit  it  to  the  Services,  OJCS,  and  OSD.  This  report 
will  be  updated  annually  and  will  provide  a  synopsis  of  JT&Es  completed  or 
ongoing  as  well  as  all  known  uses  of  JT&E  products.  Users  will  be  asked  to 
assist  in  updating  this  report. 
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b.  Management  Evaluation  Process 

All  JT&E  reports  submitted  by  the  JTD  and  the  Support  Agent 
will  be  evaluated  and  approved  by  the  DOTE.  These  reports  will  also  be 
reviewed  by  the  Senior  Advisory  Council,  and  necessary  recommendations  for 
improving  the  JT&E  process  will  be  made. 

5.  Summary 

The  JT&E  process  described  above  is  the  framework  of  an  evolving 
system  which  will  allow  timely  nomination  and  effective  planning,  program¬ 
ming  and  budgeting  of  proposed  JT&Es.  Further,  since  it  is  designed  to 
assure  early  cpnsideration  of  proposed  JT&Es,  associated  resource  problems 
should  be  minimized. 
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CHAPTER  II 
ORGANIZING  FOR  JT&E 

A.  RESPONSIBILITIES  OF  KEY  PARTICIPANTS 

1 .  Introduction 

Before  guidelines  for  organizing  a  Joint  Test  Force  can  be  dis¬ 
cussed,  an  understanding  of  the  key  participants'  responsibilities  and 
organizational  relationships  is  essential.  The  sections  which  follow 
contain  summaries  of  the  responsibilities  of  the  Director  Defense  Test  and 
Evaluation  (DOTE),  the  Office  of  the  Joint  Chiefs  of  Staff  (OJCS),  the 
Joint  Test  Director  (JTD),  the  Lead  Service,  the  participating  Services  and 
the  DOTE  Support  Agent.  Each  summary  is  supplemented  by  a  detailed  listing 
of  the  specific  responsibilities  of  the  individual  participants. 

2.  Director  Defense  Test  and  Evaluation 

The  Director  Defense  Test  and  Evaluation  has  overall  responsi¬ 
bility  for  joint  testing  and  evaluation.  In  coordination  with  the  OJCS  and 
the  Services,  he  selects  the  joint  tests  which  will  be  conducted.  His 
responsibilities  include  the  provision  of  funds  for  test-unique  items  such 
as  feasibility  studies,  test  designs,  data  management,  independent  evalua¬ 
tion,  and  instrumentation  which  is  peculiar  to  JT&E.  Specific  DDTE  respon¬ 
sibilities  are  as  follows: 

•  Select  JT&Es  (in  coordination  with  the  OJCS  and  Services) 

•  Prepare  plans,  programs  and  budgets  for  JT&E 

•  Fund  JT&E  unique  costs  (in  accordance  with  DoD  Budget  Guidance 
Manual  7110-1-M) 

•  Appoint  the  Lead  Service 

•  Appoint  and  charter  the  Joint  Test  Director 

9  Designate  participating  Services 

•  Approve  feasibility  studies  for  JT&E 

•  Approve  test  design  (in  coordination  with  the  OJCS,  Services,  and 
Joint  Test  Director) 
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•  Direct  an  independent  evaluation  of  test  data  when  required 
e  Establish  overall  policy  and  direction  for  JT&E  programs 

e  Chair  the  JT&E  Senior  Advisory  Council 
e  Provide  a  chairperson  for  the  JT&E  Planning  Committee. 

3.  Office  of  the  Joint  Chiefs  of  Staff 

The  Office  of  the  Joint  Chiefs  of  Staff  (OJCS)  receives  nomina¬ 
tions  from  the  CINCs  each  year.  The  OJCS  also  receives  nominations  from 
the  Services,  Defense  Agencies  and  OSD  for  those  JT&Es  which  concern  joint 
doctrinal  or  mission  issues.  The  OJCS  usually  recommends  integration  of 
JT&E  with  OJCS  exercises  whenever  possible.  The  OJCS  also  selects  the  CINC 
responsible  for  the  conduct  of  the  JT&E,  if  required.  Specific  OJCS 
responsibilities  include  the  following: 

e  Receive  JT&E  nominations  from  the  CINCs,  assign  priority  and 
forward  to  the  DOTE 

e  Receive  JT&E  nominations  concerned  with  joint  mission/doctrine, 
coordinate  and  forward  to  the  DOTE 
e  Recommend  JT&Es  which  can  be  integrated  with  JCS  exercises 

•  Select  CINC  responsible  for  JT&E  conduct,  if  required 

•  Participate  in  test  design 

•  Review  final  report 

•  Provide  a  permanent  member  to  the  JT&E  Senior  Advisory  Council 

•  Provide  a  permanent  member  to  the  JT&E  Planning  Committee. 

4.  Joint  Test  Director 

The  Joint  Test  Oi rector  is  responsible  to  the  DOTE  for  the  over¬ 
all  management  and  execution  of  a  specific  JT&E.  He  is  directly  responsi¬ 
ble  for  assuring  that  the  test  design  and  objectives  are  met  and  that  a 
final  test  report  and  a  JTD  management  report  are  published.  He  is 
responsible  for  the  management,  supervision  and  direction  of  all  Joint  Test 
Force  activities  and  resources.  Specific  JTD  responsibilities  are  as 
follows: 


Participate  in  test  design 

Manage  and  supervise  all  facets  of  the  JT&E 
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•  Ensure  that  test  conduct  accomplishes  test  objectives  as  directed 
by  the  DOTE 

•  Direct  and  supervise  JTD  staff 

•  Organize  a  functionally  effective  JTF 

•  Control  and  account  to  the  DDTE  for  OSD  funds  and  monitor  Service 
expenditures  related  to  the  JT&E 

•  Develop,  maintain,  and  update  funding  requirements  and  submit  the 
requirements  to  the  DDTE  and  the  Services,  as  appropriate 

•  Coordinate  the  integration  of  the  JT&E  with  JCS  exercises,  if 
required 

e  Coordinate  the  use  of  required  resources  (e.g. ,  forces  and 

weapons  systems)  and  facilities  (e.g. ,  range  and  maneuver  areas) 

•  Develop  plans  to  support  the  test  (e.g.,  field  test  plan, 
instrumentation  plan,  data  management  plan,  simulation  plan, 

«  logistics  support  plan) 

•  Collect  and  validate  data  from  the  JT&E 

•  Analyze  and  evaluate  data  and  report  test  results  to  the  DDTE 

•  Prepare  summary  statistics  and  findings  of  fact. 

•  Submit  interim  and  special  reports,  as  required,  to  the  DDTE 

•  Submit  a  JTF  final  test  report  and  a  JTD  management  report  to  the 
DDTE 

t  Acquire  and  control  special  instrumentation  and  other  investment 
resources  and  develop  a  plan  for  their  return  or  disposition  in 
accordance  with  established  procedures 

•  Oversee  acquisition  and  disposition  of  OSD-funded  materiel  in 
accordance  with  the  established  procedures  described  in  Chapter 
III,  Section  E-2  of  this  manual. 

5.  Lead  Service 

The  Lead  Service  is  responsible  to  the  Secretary  of  Defense  for 
ensuring  that  the  Joint  Test  Director  has  available  all  resources  necessary 
for  the  successful  administration  and  completion  of  the  JT&E.  These 

include  the  personnel  necessary  to  staff  the  JTF  as  well  as  non-test 
unique  resources  required  for  the  actual  test  conduct.  Other  participating 
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Services  also  provide  and  administer  resources  necessary  to  support  the 
JTD.  Specific  Lead  Service  responsibilities  are  as  follows: 
e  Nominate  Joint  Test  Director  to  the  DOTE 
e  Provide  resources,  facilities,  and  administrative  support 
e  Obligate  funds  necessary  to  support  the  JT&E,  excluding  test 
unique  expenses  funded  by  the  DDTE 

e  Support  test  planning,  execution,  evaluation,  and  reporting 
e  Designate  unit  responsible  for  providing  JT&E  support 
e  Designate  Service  points  of  contact  to  ensure  coordination  with 
participating  agencies. 

6.  Services 

The  Services  submit  to  the  DDTE  by  May  1  of  each  year  nominations 
of  the  JT&Es  they  believe  should  be  undertaken  in  the  next  five  fiscal 
years.  Information  forwarded  with  the  nominations  should  cover  the  areas 
shown  in  Figure  1-3  in  Chapter  I.  Nominations  of  JT&Es  which  concern 
doctrinal  or  mission  issues  must  be  submitted  through  the  OJCS  for  coor¬ 
dination  and  concurrence  before  they  are  forwarded  to  the  DDTE.  The  Ser¬ 
vices  also  identify  special  requirements  for  data  or  test  events  which  may 
be  incorporated  into  a  specific  JT&E.  A  prime  Service  responsibility  is  to 
plan,  program,  and  provide  necessary  resources  and  funds  not  provided  by 
OSD  to  support  the  JT&E  FYTP.  Specific  Service  responsibilities  are  as 
follows: 

•  Annually  nominate  to  the  OJCS  and/or  the  DDTE  joint  tests  for 

initiation  during  the  succeeding  five  fiscal  years 

•  Participate  in  the  preparation  of  the  test  design 

e  Identify  special  Service  requirements  for  data  or  special  test 

events  which  may  be  incorporated  in  the  JT&E 
e  Program  and  provide  non-test  unique  funds  and  resources  to  con¬ 
duct  the  approved  JT&Es 

e  Appoint  the  Deputy  Test  Director  in  coordination  with  the  DDTE 

e  Provide  fully  qualified  personnel  to  staff  the  Joint  Test  Force 

e  Designate  a  unit  or  agency  within  the  Service  responsible  for 

Service  coordination 
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•  Designate  a  point  of  contact  for  specific  JT&E 

•  Act  as  Lead  Service,  when  selected 

•  Procure  and/or  modify  test  items,  systems,  equipment,  and  instru¬ 
mentation  as  requested  by  the  Joint  Test  Director  and  coordinated 
by  the  Services 

•  Conduct  an  independent  evaluation  of  test  data,  if  desired 

•  Review  final  test  reports 

•  Provide  a  permanent  member  to  the  JT&E  Senior  Advisory  Council 

•  Provide  a  permanent  member  to  the  JT&E  Planning  Committee. 

7.  DOTE  Support  Agent 

If  required,  a  Support  Agent  is  selected  by  the  DDTE  to  conduct 
detailed  feasibility  studies  and  prepare  independent  evaluation  plans  and 
test  designs.  The  Support  Agent  may  also  perform  an  independent  evaluation 
of  the  test  results  for  the  DOTE.  The  organization  selected  may  be  a  DoD 
element,  a  Federal  Contract  Research  Center,  or  a  private  contractor.  The 
Support  Agent  will  not  work  directly  for  the  JTD  in  the  performance  of  his 
charter.  Specific  DDTE  Support  Agent  responsibilities  include: 

•  Conduct  feasibility  study 

•  Prepare  independent  evaluation  plan 

•  Develop  test  design  in  coordination  with  the  JTD  and  partici¬ 
pating  Services 

•  Prepare  independent  evaluation  report  for  the  ODTE. 

8.  ESTABLISHING  THE  JTD  STAFF 

1 .  Mission  and  Functions  of  the  JTD  Staff 

The  mission  of  the  Joint  Test  Director  and  his  staff  is  to  plan, 
execute,  evaluate,  and  report  the  results  of  the  assigned  JT&E.  In  dis¬ 
charging  his  mission,  the  JTD  will  report  directly  to  the  DDTE  for  test 
policy,  program  direction,  planning,  execution,  and  reporting  of  the  fund¬ 
ing  program.  Because  JT&E  is  a  mutually  supported  activity  with  the  DDTE 
and  the  participating  Services  providing  essential  resources,  the  JTD  must 
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have  the  full  support  of  the  Lead  Service  and  the  other  participating 
Services  to  ensure  the  successful  accomplishment  of  the  JT&E.  The  DOTE  is 
responsible  for  the  conduct  of  each  JT&E,  and  the  JTD  must  act  for  the  DDTE 
in  planning,  executing,  evaluating,  and  reporting  the  results  of  the  JT&E. 
Formally  assigned  responsibilities  of  the  JTD  were  listed  in  Section  A  of 
this  chapter. 

2.  JTD  Staff  Organization 

The  organization  of  a  JTD  staff  must  be  designed  with  a  clearly 
understood  flow  of  authority.  The  structure  may  vary  with  the  size  and 
scope  of  the  test.  There  are,  however,  certain  basic  functions  which  are 
common  to  all  joint  tests.  These  include  test  plans  and  operations, 
resource  management,  instrumentation,  data  management,  analysis,  logistics, 
and  administration. 

Past  experience  indicates  that  gathering  all  data  functions  in  a 
single  directorate  is  not  the  best  approach.  It  is  useful,  for  example,  to 
have  a  check  and  balance  between  the  instrumentation  group  and  the  data 
processing  group.  Further,  the  function  of  data  validation  is  closely 
related  to  the  analysis  function.  Data-related  functions  can  be  separated 
into  the  four  distinct  areas  of  instrumentation;  data  collection;  process¬ 
ing,  storage  and  manipulation;  and  analysis  and  validation. 

cigure  II-l  depicts  the  baseline  organizational  structure  for  a 
JTD  staff.  This  baseline  can  be  used  as  a  starting  point;  adjustments 
should  be  made  when  necessary.  In  the  early  stages  of  the  JT&E,  individ¬ 
uals  may  have  minimum  demands  in  their  intended  position  and  at  the  same 
time  possess  skills  which  are  usable  in  other  areas  and  functions.  For 
example,  analysts  on  board  in  the  early  stages  may  be  able  to  assist  the 
instrumentation  directorate  or  data  management  directorate.  Conversely,  in 
the  later  stages,  instrumentation  engineers  may  be  available  to  assist  in 
analysis  or  data  processing.  If  those  "attachments"  tend  to  be  long-term, 
the  JTD  should  consider  organizing  to  meet  the  needs  of  each  distinct  phase 
(i.e. ,  having  a  planning  organization  and  an  execution  organization).  In 
general,  that  should  not  be  the  case,  and  attaching  people  to  other  organi¬ 
zations  will  be  done  on  a  coordinated,  but  informal  basis. 
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Figure  II-1.  JTD  Staff  Organizational  Structure 


I.  Roles  of  the  Deputy  Test  Directors 

The  primary  responsibility  of  a  Deputy  Test  Director  is  to  sup¬ 
port  the  Joint  Test  Director  in  the  overall  direction  and  management  of  the 
test.  He  may  also  act  as  a  functional  director  of  one  of  the  key  organi¬ 
zational  elements,  e.g.,  Director  of  Plans  and  Operations,  Director  of  Data 
Management,  or  as  overall  Deputy  to  the  JTD. 

At  the  same  time,  a  Deputy  Test  Director  functions  in  an  impor¬ 
tant  secondary  role:  he  serves  as  the  senior  representative  of  and  spokes¬ 
man  for  his  Service.  In  this  capacity  he  must: 

•  Provide  Service-specific  expertise  in  resolving  personnel  and 
administrative  problems  related  to  his  Service 

•  Oversee  the  performance  evaluation  process  for  all  personnel  from 
his  Service  to  assure  that  ratings  are  representative  of  the 
performance 

•  Provide  advice  jnd  expertise  to  the  JTD  on  specific  Service 
issues  and  areas  of  interest 

•  Act  as  a  focal  point  for  arranging  Service  support  necessary  for 
the  conduct  of  the  test 

•  Provide  test  progress  and  significant  action  reports  to  his 
Service  on  a  continuing  basis. 

In  summary,  the  role  of  a  Deputy  Test  Director  is  to  fully  sup¬ 
port  the  conduct  of  the  test  as  a  member  of  the  JTF  and  as  the  focal  point 
for  all  matters  concerning  his  parent  Service.  As  such,  he  should  be 
granted  appropriate  authority  by  his  Service. 

4.  Scoping  the  Test 

The  first  task  for  the  JTD  is  to  achieve  a  clear  understanding  of 
the  scope  and  most  probable  time-line  for  the  test.  The  time-line  drives 
all  the  substantive  parts  of  the  test.  Funding  and  budgeting,  personnel 
acquisition  and  management,  test  planning,  instrumentation,  test  execution, 
and  reporting  are  all  sensitive  functions  of  the  time-line. 

It  is  particularly  important  for  the  JTD  to  make  his  own  judgment 
concerning  the  compatibility  of  the  proposed  scope  and  time-phasing.  He 
must  determine  that  the  scope  is  defined  sufficiently  so  that  a  valid 
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judgment  of  the  time  required  can  be  made.  At  this  early  stage,  the  JTD 
should  not  be  concerned  if  a  36-month  estimate  is  off  by  3  or  4  months. 
Instead,  the  JTD  should  be  attempting  to  discern  whether  the  test  clearly 
falls  into  one  of  the  following  three  natural  classes  of  time-lines  for 
JT&E:  Short  -  essentially  a  TDV/TAD  effort;  Normal  -  PCS  effort,  but  no 
personnel  replacement  problems;  Long  -  PCS,  test  length  greater  than  normal 
PCS  tour  length.  The  first  category  (short)  consists  of  those  tests  which 
are  sharply  limited  and  can  be  accomplished  in  less  than  a  year.  In  these 
cases,  the  JTD  staff  can  be  manned  on  a  TDY/TAD  basis.  (Examples  of  such 
tests  include  MAVERICK,  A-7/A-10,  and  Imaging  IR  MAVERICK).  The  second 
category  (normal)  is  the  class  of  test  which  is  more  than  one  but  less  than 
four  years.  In  this  class,  the  JTD  staff  will  be  assigned  on  a  PCS  basis. 
(Examples  include  ACEVAL/AIMVAL,  ARMVAL,  and  RABVAL).  Finally,  the  current 
trend  is  toward  tests  of  increasing  complexity  and  scope;  total  duration 
may  be  well  in  excess  of  a  normal  PCS  tour  length.  In  such  a  case,  the  JTD 
must  create  a  staffing  plan  which  accounts  for  personnel  departures  and 
replacements  (which  may  include  his  own). 

5.  Selecting  the  JTF  Location 

Another  important  question  which  merits  early  resolution  is 
determining  the  headquarters  location  for  the  JTF.  JTFs  have  been  located 
in  many  places,  but  generally  they  have  been  located: 

•  In  Washington,  D.C.  (or  nearby) 

•  At  the  home  station  of  a  Service  organization  with  management 
responsibilities  for  test  support  (for  example,  AFTEC,  Kirtland 
AFB,  NM  or  USACOEC,  Fort  Hunter  Liggett,  CA) 

•  At  the  actual  test  site  (for  example,  Nellis  AFB,  NV  or  White 
Sands  Missile  Range,  NM). 

In  at  least  one  test,  the  JTD  chose  to  establish,  on  a  PCS  basis, 
a  headquarters  in  Washington,  D.C.,  which  was  later  moved  PCS  to  the  test 
range.  Such  a  course  of  action  often  adds  considerably  to  management 
difficulties.  If  the  first  location  is  disestablished  when  the  second  is 
formed,  there  are  problems  of  maintaining  continuity.  Two  distinct  loca¬ 
tions  imply  separate  physical  facilities,  personnel  support,  and  fiscal 
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support  —  perhaps  even  support  by  two  different  Services.  PCS  moves  must 
be  planned  to  comply  with  established  OoO  and  Service  policies.  They  will 
also  impact  on  Service  budgets.  Two  separate  staffs  could  be  used;  how¬ 
ever,  this  adds  to  overhead  costs  for  the  test. 

There  is  strong  motivation  for  the  JTD  to  have  easy  access  to 

Washington,  D.C.  in  the  preliminary  phases  of  the  test  (shortly  after  his 

selection,  until  test  design  is  complete).  If  he  is  physically  in  Washing¬ 
ton,  the  JTD  will  be  better  able  to  personally  interact  with  the  DDTE,  the 
Support  Agent,  Service  Headquarters,  and  other  participants.  The  starting 
period  of  any  test  is  both  the  most  crucial  and  most  poorly  defined,  and 
the  presence  of  the  JTD  at  this  time  is  very  important. 

6.  Use  of  a  Start-Up  Task  Force 

Consideration  should  be  given  to  using  a  start-up  task  force  for 

the  early  stages  of  forming  the  JTF.  This  technique  has  been  used  with 

great  success.  First,  the  JTD  must  think  through  the  test  and  decide  which 
of  the  elements  are  likely  to  be  crucial.  He  can  then  convene  TDY/TAD  task 
forces  from  each  participating  Service  and  charge  them  with  creating  action 
plans  to  resolve  problem  areas.  For  example,  the  JTD  might  anticipate  that 
the  crucial  areas  will  be: 

•  Instrumentation 

•  Near  real  time  assessment 

•  Data  management 

•  Test  item  modification 

•  Simulation. 

The  JTD  must  vigorously  guide  the  conduct  of  the  task  forces  and 
ensure  that  JTD  efforts  are  coordinated  with  the  DDTE.  Task  forces  are  not 
seeking  to  define  the  details  of  a  final  solution.  Rather,  they  are  seek¬ 
ing  to  outline  methods  of  approach,  key  action  agencies,  bounding  guide¬ 
lines,  and  estimates  of  cost  and  schedule.  Such  early  resolution  of  test 
issues  and  problems  can  assure  an  orderly  start-up.  The  JTD  who  developed 
this  technique  used  a  few  key  people,  permanently  assigned  but  part  time, 
to  supervise  the  start-up  task  force.  He  used  some  of  the  same  people  to 
form  the  nucleus  of  his  PCS  JTD  staff  at  the  test  site.  The  JTD  remained 
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in  Washington,  periodically  visiting  the  test  site  during  test  execution. 
The  combination  of  identification  of  the  crucial  elements,  skillful  use  of 
start-up  task  forces,  and  aggressive  follow-up  techniques  should  assure 
effective  start-up  and  minimize  the  occurrence  of  future  crises. 

7.  Personnel  Management 
a.  Staffing 

Personnel  matters  will  be  a  major  concern  of  the  JTD  through¬ 
out  the  course  of  the  test.  For  the  sake  of  clarity,  the  personnel  consi¬ 
dered  here  are  assigned  PCS  or  TDY/TAD  to  the  staff  of  the  JTD.  In 
general,  the  JTD  will  not  be  as  concerned  with  the  background,  skills,  and 
other  attributes  of  test  player  force  or  data  collection  personnel.  Using 
the  scope  and  time-line  considerations  described  earlier  in  Section  B-4  of 
this  chapter  as  a  backdrop,  the  general  staffing  problems  are  addressed 
below. 

(1)  TDY/TAD  Staff 

For  the  short,  well-bounded  test,  selection  of  a  TDY/ 
TAD  staff  is  appropriate.  However,  the  JTD  should  carefully  review  the 
test  schedule  to  assure  that  test  planning,  test  execution,  reporting,  and 
wrap-up  actions  can  be  accomplished  within  the  normal  TDY/TAD  limit  of  179 
days.  If  not,  fully  qualified  personnel  must  be  phased  in  and  out  to  meet 
the  179-day  limit  or  a  reasonable  number  of  Secretarial  waivers  must  be 
obtained  to  satisfy  the  requirement  through  the  end  of  the  test.  For  the 
short  test,  TDY/TAD  staff  may  be  the  only  choice.  The  management  challenge 
faced  by  the  JTD  in  the  case  of  TDY/TAD  is  potentially  severe  because  of 
the  following: 

e  Difficulty  in  creating  a  truly  cohesive  joint  staff 

•  Allegiance  of  the  TDY/TAD  person  tending  to  remain  with  his  home 

organization 

e  Financial  and  personal  hardship  for  the  people  involved 
e  Motivational  difficulty  because  of  the  temporary  nature  of  the 
test. 
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However,  there  are  some  offsetting  positive  attributes 

of  TDY/TAD  staffing: 

e  Required  lead  time  is  short 

•  Because  of  the  short  total  involvement,  career  disruption  is  not 
perceived  to  be  a  problem. 

(2)  PCS  Staff 

Given  that  the  considerations  of  scope  and  time  phasing 
allow  a  PCS  staff,  the  JTD  should  select  that  technique.  Former  JTOs 
strongly  recommend  that  the  staff  be  PCS  and  on-site.  The  JTD  must  address 
the  technical  and  organizational  aspects  of  his  test  early  to  promptly 
justify  manpower  authorizations  and  requisition  personnel.  Lead  times 
vary,  depending  upon  many  factors,  but  twelve  months  is  a  reasonable  plan¬ 
ning  factor.  In  a  lengthy  test,  lead  time  is  a  major  consideration  for 
only  the  key  personnel,  since  the  build-up  in  the  staff  can  probably  be 
time-phased  without  impact. 

(3)  Obtaining  the  Staff 

The  JTD  must  provide  as  much  lead  time  as  possible  to 
issure  selection  and  assignment  of  the  best  qualified  persons.  Having 
carefully  considered  the  topics  discussed  earlier  in  this  manual,  the  JTD 
should  have  a  clear  idea  of  his  charter,  mission,  the  outline  of  his  staff 
organization,  and  his  headquarters  location.  Further,  if  he  is  using 
temporary  start-up  task  forces,  he  may  well  have  begun  to  identify  crucial 
elements  which  merit  close  management  attention.  Therefore,  he  is  now  in  a 
position  to  define  the  skill  and  grade  requirements,  by  Service,  of  his  key 
staff  members.  Having  defined  these  requirements  and  established  the 
justification,  he  should  provide  his  requirements  to  the  headquarters  point 
of  contact  for  each  of  the  Services  involved.  (For  points  of  contact  see 
Chapter  IV,  Section  A.)  In  addition  to  written  requests,  the  JTD  should 
brief  each  Service  regarding  his  current  view  of  the  test  and  the  role  the 
requested  personnel  will  discharge.  This  serves  two  purposes: 

e  It  is  the  most  effective  means  for  achieving  positive  response  to 
the  personnel  request. 
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•  It  provides  a  valuable  early  dialogue  on  the  substantive  aspects 
of  the  test. 

b.  Requirements  for  Key  Disciplines 

Research  for  this  manual  revealed  some  valuable  lessons 
learned  regarding  the  following  personnel  specialties  which  have  proven  to 
be  key  factors  in  past  tests:  security,  resource  manager,  contracting, 
program  planning  and  control,  frequency  allocation,  and  logistics. 

(1 )  Securi ty 

In  almost  every  JT&E,  security  is  an  extremely  impor¬ 
tant  factor.  Skilled  administrative  security  personnel  are  required  to 
provide  procedural  and  document  security  for  the  JTF.  Two  other  security 
areas  can  be  important  for  particular  tests: 

e  Operations  Security  (OPSEC)  for  any  test  using  operating  or 

development  equipment 

•  Communications  Security  (COMSEC)  for  JTF  telephonic  communica- 
tions  and  for  RF  communications  of  player  forces  during  the  test. 

(2)  Resource  Manager 

A  resource  manager  who  is  experienced  in  budgeting, 
funding,  and  cost  accounting  is  absolutely  essential  since  this  is  a  criti¬ 
cal  function,  especially  for  large  tests.  The  resource  manager  must  trans¬ 
fer  fiscal  authorizations  and  track  fund  commitments,  obligations,  and  dis¬ 
bursements.  Depending  upon  the  complexity  of  the  test,  the  resource 
manager  may  not  need  all  the  fiscal  skills  cited,  but  he  should  have  some 
knowledge  of  the  areas  listed. 

(3)  Contracting 

In  general,  it  is  not  recommended  that  the  JTD  have  a 
warranted  contracting  officer  on  his  staff.  However,  he  must  have  a  person 
conversant  with  the  Armed  Services  Procurement  Regulations  (ASPRs),  the 
language,  and  the  general  nature  of  the  procurement  process.  In  almost 
every  case,  the  JTO  will  be  funding  and  specifying  tasks  for  contracts 
administered  by  a  supporting  contracting  office.  The  JTD  will  probably  be 
the  Contracting  Officer's  Technical  Representative  (COTR)  with  responsibi¬ 
lity  for  protecting  the  Government's  interest  during  contract  performance. 
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(4)  Program  Planning  and  Control 

Program  planning  and  schedule  and  cost  control  repre¬ 
sent  key  management  tasks.  Not  all  JTDs  have  created  a  special  organiza¬ 
tional  element  to  handle  these  tasks.  In  lieu  of  a  program  control  office, 
the  responsibility  has  been  vested  in  a  deputy  or  a  functional  director. 
However  the  JTD  chooses  to  accomplish  them,  the  functions  of  program  con¬ 
trol  are  essential. 

(5)  Frequency  Allocation 

Radio  frequency  assignment  and  use  is  essential  to  many 
joint  tests.  Adequate  consideration  of  these  problems  in  the  planning  and 
execution  phases  can  avoid  environmental  impacts  and  can  enhance  test 
effectiveness  and  realism.  While  a  frequency  allocation  expert,  per  se,  is 
probably  not  required,  one  or  more  persons  on  the  staff  must  be  conversant 
with  the  language,  paperwork,  and  potential  delays/disruption  inherent  in 
the  frequency  allocation  process. 

(6)  Logistics  and  Facilities 

An  early  and  often  vexing  problem  is  the  establishment 
of  appropriate  facilities  and  support  for  the  JTD  and  his  staff.  Further, 
throughout  the  life  of  the  JTD  staff,  there  are  requirements  for  facilities 
or  services  which  may  not  be  normally  provided  by  the  Service  which  sup¬ 
ports  the  JTD  and  his  staff.  An  officer  skilled  in  the  area  of  facilities, 
services,  and  logistics  can  be  invaluable, 
c.  Administration 

The  administrative  function  of  the  JTD  staff  must  be  pre¬ 
pared  to  deal  effectively  with  the  specific  requirements  of  each  of  the 
Services  involved.  The  administrative  director  must  provide  cohesiveness 
and  consistency  to  the  functioning  of  the  JTF.  Professional  handling  of 
administrative  issues  can  be  of  significant  assistance  to  the  JTD  in  estab¬ 
lishing  the  truly  joint  nature  of  the  JTF.  Two  functions  which  affect  all 
permanent  members  of  JTF  are  pay  and  evaluation. 

(1)  Performance  Evaluations 

The  JTD  must  clearly  explain  the  rating  system,  empha¬ 
sizing  two  complementary  aspects.  First,  since  the  JT&E  is  an  important 
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joint  task,  the  rating  structure  will  reflect  the  organization  of  the  JTD 
staff,  and  evaluations  will  be  based  upon  the  ratee's  performance  in  dis¬ 
charging  the  mission  of  the  JTF.  Second,  and  equally  important,  ratings 
will  be  reviewed  by  the  DTD  of  the  ratee's  Service  so  that,  to  the  maximum 
extent  possible,  all  of  the  formal  and  informal  rating  rules  of  his  parent 
Service  will  be  scrupulously  observed.  The  rating  structure  will  be  as 
follows: 

•  The  JTO  will  be  rated  by  the  DOTE.  The  rating  will  be  endorsed 
by  his  parent  Service. 

•  The  DTD  will  be  rated  by  the  JTD  with  endorsement/review  or 
additional  ratings  provided  by  the  parent  Service  of  the  ratee. 
Note:  the  DTDs  may  have  additional  duties  as  functional 
directors. 

•  All  others  in  the  JTD  staff  will  be  rated  by  their  supervisors  in 
the  JTD  staff.  Endorsement  or  additional  rating  will  be  provided 
by  the  next  higher  level  of  supervision.  However,  the  JTD  must 
ensure  that  the  highest  level  of  endorsement  in' the  JTF  is  of  the 
ratee's  parent  Service  (normally  the  DTD  from  his  Service). 

The  performance  evaluation  system  is  depicted  below. 

RATEE 

JTD 

DTD 

JTF  Staff  Member 

Figure  I I- 2.  Performance  Evaluation  System 
(2)  Pay 

Expertise  on  pay  systems  of  each  participating  Service 
must  be  available.  If  that  expertise  is  available,  pay  and  travel  reim¬ 
bursement  should  offer  no  particular  difficulty.  The  Navy  and  Marine 
participants  will  be  paid  from  the  nearest  U.S.  Navy  Finance  Office.  Army 
or  Air  Force  personnel  will  be  supported  by  the  Consolidated  Base  Personnel 


PRIMARY  RATER  CONCURRENT  RATING/ENDORSEMENT 
DOTE  Parent  Service 

JTD  Parent  Service 

Immediate  Supervisor  DTD  from  Parent  Service 
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Office  at  the  nearest  installation  of  that  Service.  The  JTD  staff  need  not 
be  a  servicing  personnel  or  pay  office.  It  must  only  be  familiar  with  how 
to  access  and  use  the  support  structure  of  each  Service  and  how  to  directly 
interface  with  the  appropriate  Service  agency  to  solve  the  problems  of 
individual  JTF  personnel. 

8.  Contract!'  nq 

Virtually  without  exception,  the  preparation  and  execution  of  a 
JT&E  will  require  contractual  arrangements  for  the  procurement  of  equipment 
and  services.  For  example,  the  JT&E  may  require  surrogate  vehicles  to 
examine  new  systems  and  concepts  or  unique  instrumentation  to  collect 
and/or  process  data.  In  the  case  of  services,  contracting  may  be  an  effi¬ 
cient  and  cost-effective  method  of  acquiring  special  skills  and  expertise 
in  technical  areas  such  as  engineering,  statistics,  analysis,  and  computer 
science.  Contractual  services  during  the  initial  planning  stages  may  be 
particularly  useful  in  the  accomplishment  of  thorough  field  test  planning 
and  systems  integration. 

As  soon  as  the  JTD  is  selected,  he  should  examine  his  mission, 
organization,  and  available  resources.  In  this  examination,  the  JTD  must 
identify  the  most  crucial  contracting  areas.  Many  items  require  a  long 
lead  time;  in  order  to  have  such  items  available  in  time  for  testing, 
contracts  must  be  executed  in  a  timely  manner. 

The  laws,  regulations,  and  policies  governing  the  procurement  of 
equipment,  supplies,  and  services  are  many  and  complex.  It  is  imperative 
that  the  JTD  understand  the  procurement  process  to  effectively  discharge 
his  responsibilities.  The  Armed  Services  Procurement  Act  states  the  pre¬ 
ferred  methods  of  procurement.  The  Act  provides  for  the  use  of  the  type  of 
contract  that  best  adapts  to  the  unique  needs  of  the  JTD.  The  JTD  must 
depend  on  a  Contracting  Officer  (Procurement  Office),  duly  appointed  with 
authority  to  enter  into  and  administer  contracts.  The  JTD,  or  his  desig¬ 
nated  representative,  will  be  the  Contracting  Officer's  Technical  Repre¬ 
sentative  (COTR)  whose  primary  role  is  to  protect  the  interest  of  the 
Government  during  contract  performance. 
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While  all  OoO  and  Service  procurement  offices  operate  in  accord¬ 
ance  with  Federal  Procurement  Regulations  and  Armed  Service  Procurement 
Regulations,  their  authorities  and  operations  vary.  The  JTD  must  provide 
'he  Contracting  Officer  with  a  comprehensive  description  of  the  systems, 
equipment,  or  services  to  be  procured.  The  JTD  can  suggest  the  type  of 
contract  based  on  his  specific  needs.  Formal  advertisement  is  the  pre¬ 
ferred  method  of  procurement;  however,  negotiated  purchases  are  authorized 
where  circumstances  require  or  justify  a  departure  from  formal  advertising. 
Prominent  factors  in  such  cases  are  that  often  the  procurements  are  of  a 
unique  character  or  time  constraints  preclude  lengthy  formal  procurements. 
In  some  cases,  contracts  may  exist  which  can  be  modified  to  provide  the 
required  equipment  or  service  and  maximize  the  JTD's  ability  to  meet  the 
established  schedule.  Early  development  of  the  procurement  requirements 
for  equipment  and  services  and  prompt  selection  of  a  Contracting  Office 
which  can  best  serve  the  needs  of  the  JTD  is  vital  to  successful  accomplish¬ 
ment  of  the  JTD's  mission. 


9.  Financial  Management 

Sound  financial  management  is  essential  if  the  test  objectives 
are  to  be  achieved  within  the  bounds  of  the  resource  limits  established  by 
the  DOTE  and  the  Services.  Guidance  concerning  the  development  and  imple¬ 
mentation  of  a  financial  management  system  is  contained  in  Chapter  III. 


CHAPTER  III 
JT&E  IMPLEMENTATION 


A.  OVERVIEW 

1.  Introduction 

This  section  provides  an  overview  of  the  general  process  of  joint 
test  execution.  In  the  final  analysis,  the  JTD  is  the  individual  respon¬ 
sible  for  the  proper  execution  of  the  joint  test.  It  is  the  JTD  who  must 
assemble  the  organization  and  resources  to  conduct  the  test  and  ensure  that 
they  are  used  in  a  manner  which  provides  meaningful  results  consistent  with 
the  JTF  charter  and  test  objectives.  However,  the  JTD  does  not  operate  in 
isolation;  the  DOTE,  the  Lead  Service,  the  participating  Services,  and  the 
Support  Agent  all  have  roles  to  play  in  the  execution  of  a  joint  test. 
Thus,  the  JTD  must  be  aware  of  these  Agencies'  responsibilities  and  use 
then  to  ensure  that  the  test  is  accomplished  effectively  and  economically. 

2.  Structure  of  Testing 

Generally,  any  test  can  be  viewed  in  terms  of  four  components: 
planning,  execution,  analysis,  and  post-test  activities. 

a.  Planning 

Strictly  speaking,  joint  test  planning  begins  many  months 
prior  to  the  appointment  of  the  JTD  and  the  formation  of  the  JTF.  However, 
planning  is  a  continuous  process;  consequently,  the  JTD  should  become 
thoroughly  familiar  with  all  program  documentation  and  be  prepared  to 
become  fully  involved  in  test  planning  regardless  of  the  stage  at  which  he 
is  appointed.  Comprehensive  planning  for  a  joint  test  requires  that  the 
test  be  viewed  as  a  total  program,  not  just  a  collection  of  discrete  acti¬ 
vities.  This,  in  turn,  requires  thorough  familiarity  with  the  feasibility 
study,  involvement  in  preparation  of  the  test  design,  and  positive  control 
over  the  development  of  the  Field  Test  Plan,  the  Instrumentation  Plan,  the 
Oata  Management  Plan,  the  Logistics  Support  Plan,  and  the  Simulation  Plan, 
if  one  is  required. 
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In  many  cases,  lengthy  and  complex  JT&Es  will  overlap  with 
other  test  or  exercise  activities  or  conflict  with  operational  require¬ 
ments.  The  JTD  must  coordinate  his  planning  activities  carefully  to  avoid 
schedule  conflicts.  Conflicts  which  cannot  be  solved  by  the  JTD  will  be 
resolved  by  the  DOTE. 

b.  Execution  * 

Test  execution  is  the  measure  of  the  adequacy  of  test 
planning.  While  heroic  efforts  have  been  attempted  to  change  the  course  of 
past  tests  at  the  mid-point,  no  amount  of  analysis  can  recover  data  that 
was  not  collected  or  data  that  was  invalid  in  its  content.  The  JTD  will  be 
the  sole  authority  for  implementing  the  test  plan;  directing  test  resources 
so  that  valid  data  is  generated  to  meet  test  objectives;  and  ensuring  that 
the  data  is  collected,  reduced,  and  managed  to  facilitate  effective 
analysis. 

Test  execution  consists  of  two  phases:  preparation  for 
testing  and  active  testing.  Test  preparation  involves  scheduling  and 
assembling  required  resources,  training  test  players,  verifying  test 
support  from  other  agencies  (including  range  facilities  and  contractors), 
and  ensuring  that  approved  data  management  procedures  are  implemented  and 
operating. 

Active  testing  is  the  phase  during  which  the  basic  mission 
of  the  JTF  is  fulfilled.  As  a  percentage  of  the  overall  life  of  a  JTF, 
active  testing  may  be  quite  short.  However,  the  phase  will  be  intense, 
requiring  day-to-day,  hour-by-hour  management  by  the  Joint  Test  Director. 
Critical  aspects  of  active  testing  are  test  control,  data  management,  and, 
as  a  goal,  near  real-time  assessment.  Since  the  primary  function  of  test 
execution  is  the  collection  of  data  to  support  the  test  objectives,  the  JTD 
must  know  that  the  test  is  progressing  according  to  plan,  that  data  are 
being  captured  as  planned,  and  that  the  results  are  useful. 

c.  Analysis 

The  nature  of  joint  testing  requires  that  the  JTD  analyze 
the  data  generated  during  test  execution  and  that  he  submit  a  test  report 
containing  an  evaluation  of  test  results.  The  JTD  is  the  best  qualified  to 
determine  the  validity  of  the  data,  understand  the  details  of  each  trial, 
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and  thoroughly  comprehend  the  test  from  beginning  to  end.  Analysis  goes 
beyond  simply  reducing  and  storing  data  for  later  manipulation  in  an 
independent  evaluation.  Often,  the  analysis  performed  will  be  statistical 
in  nature  in  an  attempt  to  accept  or  reject  hypotheses  stated  in  the  test 
objectives.  Analysis  should  therefore  be  an  ongoing  feature  (near  real 
time)  of  test  management  rather  than  a  post-test  activity,  since  it  is 
likely  that  the  conduct  of  specific  trials  during  the  test  can  be  of  great 
assistance  or  hindrance  in  drawing  inferences.  Analysis  efforts  should  be 
integrated  so  that  those  who  are  involved  in  the  test  planning  process  also 
take  part  in  the  analysis  of  the  data  collected, 
d.  Post-Test  Activities 

Post-test  activities  include  test  reporting  and  briefing, 
disposition  of  resources,  and  the  establishment  of  a  data  repository.  At 
the  end  of  a  joint  test,  two  major  reports  are  prepared.  The  first  is  the 
Test  Report  which  documents  the  factual  aspects  of  the  test  as  well  as  the 
JTD's  analysis  and  evaluation  of  test  results.  The  other  report  is  a  Man¬ 
agement  Report  to  document  the  JTD's  assessment  of  the  test  objectives, 
lessons  learned,  recommendations,  and  the  potential  utility  of  the  test. 

B.  TEST  PLANNING 

1 .  Role  of  the  Joint  Test  Director  in  Test  Planning 

Joint  test  planning  begins  when  the  test  is  first  nominated. 
Each  nomination  is  accompanied  by  a  preliminary  test  concept  containing  a 
brief  description  of  the  test  purpose,  objective,  concept  and  utility, 
along  with  a  budget  estimate.  After  a  test  is  selected,  the  Planning 
Committee  coordinates  the  preparation  of  an  amplification  of  the  prelimi¬ 
nary  test  concept  for  submission  to  the  DOTE  for  use  in  preparing  the 
Feasibility  Study,  the  Test  Design,  and  the  Evaluation  Plan. 

The  JTD  should  be  selected  and  the  JTF  organized  as  early  as 
possible  to  allow  participation  in  development  of  the  Test  Design.  The  JTD 
will  actively  provide  input  and  coordinate  with  the  DOTE,  the  participating 
Services,  and  the  Support  Agent  at  every  opportunity,  since  the  results  of 
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the  Test  Design  and  requirements  of  the  Evaluation  Plan  will  be  the  basic 
building  blocks  of  the  Field  Test  Plan  he  will  have  to  develop.  Conse¬ 
quently,  the  JTD  should  identify  issues  and  problems  which  would  limit  his 
ability  to  plan  for  and  manage  the  test.  The  Test  Design  should  have  the 
full  confidence  of  the  JTD  and  participating  Services  or  Agencies;  the  JTD 
should  not  proceed  until  he  has  an  acceptable  Test  Design. 

Following  the  completion,  coordination,  and  approval  of  the  Test 
Design,  the  JTD  assumes  responsibility  for  accomplishing  the  remainder  of 
the  test  planning  process.  He  will  prepare  the  Field  Test  Plan  and  will 
constantly  review  and  revise  it  as  necessary  as  the  test  proceeds.  Since 
the  Field  Test  Plan  is  based  upon  the  Feasibility  Study,  Test  Design,  and 
Evaluation  Plan,  it  is  essential  that  the  JTD,  if  he  has  not  participated 
in  their  development,  be  thoroughly  familiar  with  their  content  and  impli¬ 
cations.  The  next  sections  summarize  the  contents  of  these  documents.  The 
following  sections  then  discuss  the  development  of  the  Field  Test  Plan,  the 
Data  Management  Plan,  the  Instrumentation  Plan,  the  Logistics  Management 
Plan,  and  the  Simulation  Plan. 

2.  Feasibility  Study 

The  Feasibility  Study  should  describe:  (1)  the  test  concept; 
(2)  the  general  test  objectives;  (3)  a  preliminary  schedule;  and  (4)  an 
estimate  of  the  resources  required.  The  JTD,  if  appointed,  and  the  par¬ 
ticipating  Services/Agencies  should  assist  the  DDTE  Support  Agent  in 
conducting  the  study  and  should  also  review  the  final  report. 

In  reviewing  the  general  test  objectives,  the  JTD  and  the 
Services  should  be  concerned  with  translating  them  into  meaningful  test 
elements.  A  determination  must  be  made  concerning  the  achievabil ity  of  the 
test  objectives  relative  to  the  available  resources. 

A  careful  review  of  the  preliminary  schedule  is  essential.  There 
is  a  real  possibility  that  the  Feasibility  Study  will  be  based  on  an 
optimistic  schedule  which  does  not  adequately  consider  actual  test  problems 
(e.g. ,  late  equipment  delivery,  mission  failures,  conflicts  with  other  test 
or  operational  requirements,  instrumentation  problems,  weather  at  the  test 
site).  If  the  schedule  is  not  achievable,  the  JTD  and  the  participating 
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Services  should  develop  a  more  suitable  schedule  and  submit  it  to  the  DDTE 
for  approval.  The  DOTE  is  responsible  for  establishing  the  priority  for 
test  accomplishment. 

Similarly,  the  estimate  of  resource  requirements  must  be  care¬ 
fully  scrutinized.  Often,  feasibility  studies  implicitly  assume  that 
required  resources  will  be  made  available.  The  JTD  and  the  Services  should 
identify  resources  such  as  instrumentation  or  simulators  which  are  likely 
to  be  difficult  to  obtain  on  schedule  (or  at  all),  and  determine  the  impact 
of  the  lack  of  such  items  on  the  JTD's  ability  to  meet  the  test  objectives. 
Serious  issues  which  arise  as  a  result  of  this  review  should  be  raised  with 
the  ODTE  and  the  Support  Agent  for  resolution  as  quickly  as  possible. 

3.  Test  Design 

The  Test  Design  will  provide  specific  test  objectives,  test 
events,  instrumentation,  test  methodology,  data  requirements,  data  manage¬ 
ment  needs,  and  analysis  requirements.  Ideally,  the  JTD  should  be  con- 
tinously  involved  with  the  Support  Agent  during  the  development  of  the  Test 
Design  because  of  the  considerable  impact  it  will  have  on  his  test  planning 
and  execution.  The  Test  Design  should: 

•  Provide  a  refined  estimate  of  resources  required 

•  Structure  and  organize  the  approach  to  testing  in  terms  of 

specific  test  objectives 

•  Identify  key  measures  of  effectiveness  (MOE) 

•  Identify  the  number  and  type  of  test  events 

•  Identify  dataT "required  and  how  the  data  will  be  gathered,  stored, 
and  analyzed 

•  Recognize  special  requirements  of  participating  Services/Agencies 

•  Provide  sufficient  flexibility  to  allow  for  the  unexpected 

•  Be  coordinated  with  the  participating  Services  and  approved  by 

the  DOTE. 

The  JTD  should  work  toward  ensuring  that  these  requirements 
are  met  in  a  way  which  facilitates  his  job  of  conducting  the  test. 

Understanding  the  key  elements  of  a  Test  Design  and  being  prepared  to  pro¬ 
vide  input  to  the  Support  Agent  are  essential  to  accomplishing  this  goal. 
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Listed  below  are  some  of  the  key  elements  along  with  suggestions  concerning 
areas  in  which  the  JTD  should  be  especially  alert: 

a.  Test  Approach 

The  test  approach  specifies  the  methods  and  techniques  to  be 
used  in  conducting  the  test.  The  JTD  should  carefully  review  with  the 
participating  Services  the  assumptions  concerning  test  locations, 
resources,  and  time  requirements  underlying  the  approach. 

b.  Constraints  and  Limitations 

The  JTD  should  be  in  agreement  with  the  Support  Agent's 
perception  of  constraints  and  limitations  which  bound  the  test.  The  JTD's 
previous  operational  and/or  testing  experience  should  be  applied.  Typical, 
but  often  overlooked,  constraints  include:  instrumentation,  a  realistic 
test  scenario,  test  schedule  compression,  range  safety,  lead  time  for  test 
items,  availability  of  support  equipment,  and  priority  conflicts  on  the 
test  range. 

c.  Identification  of  Required  Data  Elements 

The  data  elements  required  to  meet  test  objectives  determine 
the  types  of  measurements  to  be  made  and  the  instrumentation  required  to 
obtain  them.  Qualitative  information  may  be  required,  thus  necessitating 
development  of  structured  questionnaires  and  debriefings.  The  JTD  should 
thoroughly  understand  the  required  data  elements  in  order  to  properly  plan 
for  their  collection,  reduction,  and  use  in  analysis. 

d.  Automation 

The  JTD  must  trade-off  the  desire  to  automate  data  collec¬ 
tion  with  the  cost  of  such  automation.  For  one-of-a-kind  tests  wit.h  few 
repetitions,  it  is  usually  preferable  to  rely  on  manual  data  processing 
rather  than  develop  complex  computer  software.  On  the  other  hand,  few  new 
JTDs  realize  the  amount  of  data  which  can  be  generated  in  a  large  JT&E.  In 
one  test,  over  fifty  cubic  feet  of  "data"  were  estimated  to  have  been 
collected  each  day. 

e.  Statistical  Design 

Most  test  designs  are  based  on  a  statistical  model  which 
allows  generalization  from  relatively  few  observations.  The  complexity  and 


extent  of  testing  hinges  on  the  type  of  statistical  model  being  used.  The 
JTD  should  become  familiar  with  the  statistical  bas-is  for  the  Test  Design 
as  well  as  the  trace-offs  between  the  number  of  data  points  actually 
obtained  in  the  test  and  the  amount  of  uncertainty  involved  in  generalizing 
from  the  test.  A  discussion  of  statistical  analysis  is  provided  in  Section 
D  of  this  chapter.  A  statistician  or  operations  researcher  must  be  part  of 
the  JTF. 

It  should  be  remembered  that  statistical  models  are  based  on 
underlying  assumptions  about  the  "real  world,"  and  that  a  good  operational 
scenario  may  conflict  with  the  assumptions  for  a  specific  statistical 
design.  An  alternative  design  may  be  more  consistent  with  the  scenario  and 
testing  limitations;  consequently,  the  JTD  should  be  prepared  to  not  only 
critique  the  Test  Design  on  this  point  but  to  offer  alternatives  as  well. 
The  statistical  design  will  drive  the  number  of  repetitions  required  of 
each  test  event.  Consequently,  it  has  a  considerable  impact  on  test  costs 
and  realism. 

f.  Resource  Estimate 

It  is  essential  that  the  test  design  specify  all  facilities, 
manpower,  equipment,  and  instrumentation  required  from  the  Services.  This 
allows  for  maximum  planning  and  budgeting  time  and  will  materially  reduce 
future  conflicts  and/or  limitations. 

4.  Evaluation  Plan 

The  Evaluation  Plan  is  closely  linked  to  the  Test  Design,  espe¬ 
cially  the  statistical  model(s)  on  which  the  Test  Design  is  built.  Since 
the  JTD  will  be  expected  to  provide  evaluation  and  analysis  of  test  data, 
he  must  understand  the  Evaluation  Plan  well  enough  to  ensure  that  his  own 
evaluations  and  analyses  are  consistent  with  it.  Further,  the  JTD  should 
understand  that  any  changes  made  to  the  Test  Design  will  have  to  be 
reconciled  with  the  Evaluation  Plan.  For  example,  if  the  Test  Design  is 
built  on  a  distribution-free  (nonparametric)  model,  then  the  Evaluation 
Plan  cannot  call  for  regression  analysis  on  certain  data,  since  regression 
assumes  a  certain  distribution  to  underlie  the  population  being  tested. 
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Other  aspects  of  the  Evaluation  Plan  requiring  scrutiny  by  the 
JTD  are  the  assumptions  about  resources  and  level-of-effort  required  to 
conduct  the  analysis.  Oata  analysis  efforts  are  often  seriously  under¬ 
estimated.  For  example,  one  JT&E  subtest  took  three  weeks  for  testing  and 
required  two  person  years  of  analytical  effort.  The  operations  researcher 
or  statistician  on  the  JTF  will  have  expert  knowledge  of  how  to  scope  the 
required  level  of  effort. 

5.  Field  Test  Plan 
a.  Introduction 

The  test  plan  translates  the  test  design  into  "real  world" 
resources,  procedures,  and  responsibilities.  The  JTD  has  full  responsi¬ 
bility  for  preparing  this  document;  consequently,  it  provides  him  with  the 
best  opportunity  to  insert  into  the  JT&E  planning  process  his  own  methods 
of  accomplishing  test  objectives  within  the  constraints  of  time,  budget, 

manpower,  and  material  resources.  In  preparing  the  Field  Test  Plan,  the 

* 

JTD  should  remember  that,  even  deep  into  the  test,  planning  remains  a 
continuous  process  involving  ongoing  studies,  analyses,  validations, 
reviews,  and  changes. 

Experience  indicates  that  operational  realism  has  a  positive 
effect  on  the  validity,  credibility,  and  utility  of  joint  testing.  Major 
characteristics  of  realistic  tests  are: 

•  Operational  and  support  personnel  with  relevant  combat  experience 
should  participate  in  the  selection  of  test  objectives  and 
development  of  the  test  design.  They  should  also  be  participants 
in  data  collection  and  evaluation  (White  Force)  functions. 

•  System  operators  should  not  have  test  data  collection  responsi¬ 
bilities;  this  function  should  be  the  responsibility  of  trained 
White  Force  personnel  so  that  system  operator  performance  is  not 
affected. 

t  Real-time  casualty  assessment  and  damage  assessment  should  be 
considered  if  they  can  be  done  accurately  and  will  contribute  to 
the  accomplishment  of  the  objectives. 
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•  Instrumentation  should  be  provided  which  permits  the  operators 
and  the  threat  to  exercise  (by  simulation)  options  available  to 
them  in  combat. 

•  Instrumentation  should  be  as  invisible  as  possible  and  should  not 
be  cumbersome  or  impose  limitations  on  the  systems,  equipment,  or 
personnel  in  the  test.  Realism  should  be  a  key  factor. 

•  The  program  should  consist  of  exercises  simulating  combat  or 
combat  support  situations  in  which  test  data  collection  is  super¬ 
imposed.  They  may  be  supplemented  by  experiments  and  other 
measurements  not  involving  exercises. 

•  Exercises  should  be  sized  for  simulations  of  two-sided 

engagements . 

•  Operators  of  systems  being  evaluated  should  be  subjected  to 
combat  zone  conditions  and  status  during  the  test. 

•  Test  players  should  be  subjected  to  stress  as  in  combat,  using 
isolation,  rumors,  noise,  odors,  smoke,  pyrotechnics,  blank 
charges,  time  constraints,  foreign  languages,  and  other  devices 
and  procedures. 

•  System  operators  should  be  typical  operational  personnel  with 
varying  skill  levels  (including  system  maintenance  personnel)  and 
should  use  approved  and  established  operational  procedures. 

•  Tactical  profiles  and  combat  delivery  modes  should  be  used. 

•  Testing • should  be  performed  in  a  variety  of  operationally  repre¬ 

sentative  physical  environments,  including  climate  and  season, 
weather,  terrain,  vegetation,  visibility,  and  electronic  warfare 
characteristics. 

e  Scenarios  should  provide  planned  conditions  in  which  crews  react 
to  "enemy"  actions  and  system  problems. 

e  Uncertainty  must  be  maintained  in  the  minds  of  the  players  if 

valid  data  are  to  be  collected  for  such  measurements  as  detection 
times,  detection  probabilities,  reaction  times,  engagement  oppor¬ 
tunities,  probabilities  of  success,  or  other  such  performance- 
related  measurements. 
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•  Threats  should  be  simulated  according  to  current  OIA-approved 
intelligence  on  enemy  doctrine,  deployments,  tactics,  appearance, 
signals,  and  actions. 

e  Tactics  should  be  consistent  with  Joint  and  Service  doctrine,  the 
selected  military  situation,  and  the  test  environment. 

e  Whenever  possible,  it  is  desirable  to  avoid  distortion  of  event 
results  which  can  occur  when  small  numbers  of  player  force  per¬ 
sonnel  are  involved  in  repetitious  tasks. 

The  following  guidelines  are  provided  to  assist  the  JTD  in 
scoping  his  Field  Test  Plan: 

e  Understand  and  attempt  to  resolve  test  issues  early 

•  Identify  the  most  promising  testing  opportunities  and  potential 
problem  areas  which  will  impact  the  test 

•  Explore  alternative  courses  of  action  for  accomplishing  the  test 

•  Assemble  the  best  available  talent  and  know-how  for  planning  the 
test 

•  Establish  and  adhere  to  a  structured  planning  process 

e  Initial  resources  commitments  are  based  on  the  test  design. 
Early  and  thoughtful  justification  are  absolutely  essential  when 
the  field  test  plan  changes  affect  resources  requirements, 
b.  Basic  Components  of  the  Field  Test  Plan 

The  size  and  complexity  of  the  Field  Test  Plan  are  dictated 
by  the  nature  of  the  particular  joint  test  being  planned.  Nevertheless, 
the  Test  Plan  should  address  a  minimal  set  of  issues  to  be  a  useful, 
"living"  document.  This  section  discusses  the  essential  components  of  the 
Field  Test  Plan. 

(1)  Test  Purpose  and  Objectives 

This  section  provides  a  brief  statement  of  the  overall 
purpose  of  the  test.  Critical  issues  relevant  to  the  test  are  also  identi¬ 
fied.  Finally,  the  general  and  specific  objectives  of  the  test  should  be 
clearly  stated  with  priorities  assigned. 
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(2)  Concept  of  Test  Operations 

This  section  defines  all  aspects  of  the  system/concept 
to  be  tested  including:  test  items,  operator/training  requirements,  main¬ 
tenance  concepts,  personnel  and  training,  logistics  and  supply,  scenario 
requirements,  expected  test  duration,  and  special  supporting  personnel  and 
equipment. 

(3)  Scenario 

As  soon  as  the  initial  concepts  of  the  test  are 
approved,  the  JTO  should  have  a  detailed  scenario  "wargamed"  by  the  staff. 
While  this  scenario  will  probably  change,  it  prevents  early  misconceptions 
from  becoming  embedded  in  resource  estimates.  The  individual  Services  can 
also  become  more  actively  involved  by  participating  in  the  scenario  review 
process  rather  than  simply  awaiting  the  draft  test  plan.  The  JTD  should 
carefully  consider  the  requirement  to  employ  multiple  test  scenarios.  Some 
evaluations  are  likely  to  be  very  scenario-sensitive.  But  costs  of  mul¬ 
tiple  scenarios  must  be  a  prime  consideration. 

(4)  Method  of  Accomplishment 

This  section  defines  how  each  test  or  subtest  is  to  be 
accomplished.  In  general,  the  JTO  should: 

e  Identify  objectives  for  each  test/subtest 

e  Describe  the  actions  to  be  taken  and  procedures  to  be  used  during 
the  conduct  of  each  test/subtest 
e  Summarize  raw  data  to  be  collected  and  their  sources 

e  Describe  data  collection  methods  an(f  instrumentation  requirements 

e  Summarize  all  test  events  necessary  to  accomplish  each  test/ 
sub test 

•  Prepare  trial/variable  matrices  as  appropriate. 

(5)  Test  Schedule 

The  JTD  should  specify  significant  milestone  dates, 
using  a  line  chart  (if  appropriate)  showing  duration  and  time  phasing  of 
Important  test  activities.  Example  milestone  dates  include: 
e  Distribution  of  Field  Test  Plan 

e  Finalization  of  support  agreements 
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•  Equipment  and  instrumentation  procurement  or  modification 

•  Briefings  and  meetings 

•  Contractor  support  procurement 

•  Deployment  of  supporting  units 

•  Test  commencement 

•  Test  completion. 

(6)  Test  Management  and  Organization 

In  this  section,  the  JTD  identifies  key  personnel  and 
command  relationships.  In  addition,  this  section  specifies  points  of 
contact  within  the  JTF  and  within  each  Service  supporting  the  test. 
Finally,  the  organization(s)  exercising  administrative  support  for  JTF 
personnel  should  be  identified.  The  JTD  should  execute  a  clear  Memorandum 
of  Understanding  (MOU)  with  the  Lead  Service  and  each  participating 
Service/Agency. 

(7)  Responsibilities 

This  component  of  the  Field  Test  Plan  identifies  each 
participating  organization  and  lists  their  responsibilities  along  with  the 
services  and/or  equipment  they  are  to  provide.  Much  of  this  information  is 
drawn  from  the  directives  issued  by  the  DOTE;  however,  amplification  of 
these  directives  will  be  required  to  avoid  misunderstandings  during  the 
test. 

(8)  Personnel 

This  section  identifies  the  personnel  (in  addition  to 
the  JTF)  required  to  support  the  test  program  in  terms  of:  job  title, 
grade,  specialty,  period  of  utilization,  providing  organization,  and  number 
required.  The  Lead  Service  and/or  the  designated  Service  Agency  will 
provide  the  JTD  with  assistance  in  formulating  these  requirements.  In 
addition,  this  section  should  specify  any  specialized  training  which  test 
participants  will  require.  This  is  an  important  consideration  for  tests 
involving  new  doctrine/tactics  or  unusual  data  collection/instrumentation 
devices. 
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(9)  Required  Test  Reports 

This  section  of  the  plan  states  the  required  frequency 
of  test  reports,  their  recipients,  special  content/format  instructions,  and 
preparation  responsibilities.  At  a  minimum,  the  JTD  should  identify  and 
schedule  the  Interim  Report,  Management  Report,  and  Final  Test  Report. 

(10)  Safety 

Safety  planning  will  vary  from  test  to  test  based  on 
inherent  hazards.  At  a  minimum,  this  section  should:  designate  a  Safety 
Officer  and  outline  his  responsibilities,  identify  critical  safety  aspects 
of  the  test,  specify  procedures  to  be  followed,  and  establish  an  emergency 
reaction  plan. 

(11 )  Security 

This  section  should  discuss  all  security  aspects  of  the 
test  including  storage,  handling  procedures,  restricted  areas,  TEMPEST,  and 
communications  security.  The  JTD  will  develop  a  detailed  classification 
guide  consistent  with  OoD/Service  requirements  and  directives. 

(12)  Environmental  Protection 

If  an  environmental  assessment  has  shown  the  test  to 
have  potential  impact  upon  the  environment,  this  section  should  identify 
those  actions  and  procedures  which  will  be  taken  to  minimize,  mitigate,  or 
neutralize  any  adverse  effects  of  the  test.  This  will  normally  be  subsumed 
in  the  Environmental  Impact  Statement  for  the  range  or  test  facility  to  be 
used. 

c.  Coordination  and  Approval 

The  completed  Field  Test  Plan  is  submitted  to  participating 
Services,  the  Lead  Service,  and  the  DDTE  for  review  and  concurrence.  Sub¬ 
mission  for  approval  is  made  to  the  DDTE. 

d.  Other  Components  of  the  Field  Test  Plan 

Each  joint  test  is  unique.  Consequently,  the  Field  Test 
Plan  will  contain  additional  components  to  reflect  this  uniqueness. 
Usually,  these  additions  will  be  in  the  form  of  annexes  to  the  basic 
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document  outlined  above.  Examples  of  unique  test  aspects  requiring  addi¬ 
tional  amplification  include: 

a  Extensive  use  of  computer  simulation  as  part  of  the  testing 
process 

a  Specialized  simulators 

e  Extensive  equipment  modification. 

In  addition  to  dealing  with  unique  aspects  of  particular 
tests,  annexes  should  be  provided  to  discuss  in  detail  certain  management 
problems  which  will  arise  in  any  test.  Such  annexes  include: 
e  Test  Design  Overview 

e  Reliability  and  Maintainability  Data 

e  Intelligence/Threat  Information. 

6-  Data  Management  Plan 

a.  Purpose 

The  Data  Management  Plan  is  a  formal  document  which  provides 
the  complete,  detailed,  and  integrated  procedure  for  the  time-phased  task 
of  identifying,  collecting,  processing,  analyzing,  reporting,  and  disposing 
of  data  which  are  relevant  to  the  objectives  of  the  joint  test.  The  data 
management  process,  which  is  depicted  in  Figure  III-l,  is  illustrative  of 
the  logic  incorporated  in  test  design.  The  process  involves  six  distinct 
f uncti ons : 

e  Identification  of  the  required  data  elements  and  sources  and 

reli ability/confidence  assessment  of  data  elements  and  sources 

e  Response  collection  to  include  design  of  forms  and  questionnaires 
e  Data  processing  and  reduction  to  include  verification  of  data 
quality  and  reduction  algorithms 
e  Data  analysis  and  assessment 

e  Report  processing 

e  Oata  disposition. 

b.  Content 

At  a  minimum,  the  Data  Management  Plan  must: 
e  Detail  the  flow  of  processing  of  data 

e  Outline  specific  responsibilities  for  collection  and  management 


of  data 
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Figure  III— 1 .  Data  Management  Process 
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•  Determine  data  collection  procedures  which  require  specialized 
training 

e  Provide  data  formats 

e  Estimate  the  amounts  and  types  of  data 
e  Specify  required  measurement  accuracies 

e  Provide  for  the  identification  and  validation  of  additional' 
measures  of  effectiveness 

•  Provide  a  plan  for  data  storage  and  retrieval 

•  Size  the  data  management  system 

e  Provide  formal  procedures  for  data  quality  control. 

Identification  and  protection  of  data  are  primary  requisites 
of  the  data  management  system.  Provisions  must  be  made  for  prompt  labeling 
and  dating  of  collected  data.  Back-up  copies  must  be  made  of  all  data. 
The  flow  of  data  should  be  directed  by  routing  instructions  which  indicate 
where  the  data  will  go,  who  will  process  it,  and  which  specific  analysis  or 
evaluation  is  to  be  performed.  In  addition  to  the  normal  data  flow,  a 
capability  must  be  provided  for  quick  look  analysis  as  a  check  against 
invalid  data  or  unanticipated  difficulties.  All  data  flows  should  be 
through  a  single  data  control  function  charged  with  the  responsibilities  of 
protection,  quality  control,  copying,  and  archiving, 
c.  Software  Development 

To  produce  an  effective  automated  data  management  system,  a 
software  development  plan  must  be  adopted  and  followed  in  a  systematic 
manner.  Regardless  of  the  software  development  plan  chosen  it  must 
specify: 

e  Problem  definition 

e  Detailed  specifications 

e  Design  period 

e  Implementation  phase 

e  Software  test  integration 

e  Acceptance. 

Throughout  the  software  development  cycle  the  use  of  milestone  reviews  is 
recommended.  The  most  critical  data  management  task  is  the  early  detection 
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and  correction  of  slips  in  the  software  development  schedule.  Milestone 
reviews  constitute  an  excellent  technique  for  exercising  this  control. 
Delays  can  be  and  often  are  the  result  of  perturbations  in  seemingly 
unrelated  areas  of  the  project.  It  is  imperative  that  all  task  managers  be 
aware  of  all  scheduling  problems.  They  must  evaluate  and  report  on  how 
apparently  unrelated  problems  may  affect  their  particular  areas, 
d.  Lessons  Learned 

Since  a  good  Data  Management  Plan  is  critical  to  the  success 
of  a  joint  test,  some  situations  which  have  occurred  in  previous  tests  are 
.  listed  below  so  that  future  JTDs  may  benefit  from  these  "lessons  learned." 

•  The  requirements  for  manual  data  collection  and  processing  are 
always  underestimated.  Data  system  rehearsals  help  to  better 
identify  requirements. 

•  Strict  control  over  the  test  data  by  data  test  team  members 
should  be  maintained.  This  ensures  correct  handling  of  the  data 
within  the  guidelines  of  the  approved  data  management  plan. 

•  Turnaround  times  for  data  processing  should  be  established  and 
contracted  with  any  outside  agency  supporting  the  test,  such  as 
test  equipment  maintenance,  film  processing  laboratories,  and 
data  recording  tape  reloading  crews.  This  ensures  that  data  are 
collected  and  made  available  in  a  timely  manner. 

•  Quality  control  guidelines  should  be  established  for  those 
Service  Agencies  not  under  direct  control  of  the  test  team. 

•  Mission  success  evaluation  criteria  with  respect  to  effective 
versus  noneffective  mission/tests  should  be  determined  early  in 
the  test  program. 

•  Early  consideration  should  be  given  to  the  impact  of  workload  on 
an  operator's  ability  to  log  in-test/in-flight  data  notes. 

e  The  test  team  should  anticipate  data  losses  occurring  during 
early  use  of  the  data  management  system.  Therefore,  nonrepeat- 
able  test  events  should  be  scheduled  only  after  the  data  manage¬ 
ment  system  has  been  fully  exercised.  Data  system  rehearsals  are 
a  must. 
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•  Whenever  possible,  consideration  should  be  given  to  the  retention 
of  data  useful  to  other  agencies.  Outside  agency  requirements 
should  be  submitted  early  to  the  test  team. 

e  The  purpose  of  the  test  is  to  meet  test  objectives  and  not  to 
fill  out  data  forms.  If  data  forms  are  not  applicable,  delete 
their  use.  Collect  only  the  data  which  is  absolutely  necessary. 
The  collection  of  unnecessary  data  costs  money  and  time. 

e  Computer  communication,  availability,  and  reliability  should  be 
major  concerns  in  the  decision  to  use  an  automated  data  base 
management  system.  The  qu  ity  of  phone  lines  and  the  obliga¬ 
tions  of  the  computer  center  may  involve  weeks  or  months  of  lead 
time  before  the  entire  system  is  ready  for  use. 

•  Detailed  documentation  of  the  data  sources,  types,  and  formats  to 
be  used  by  the  test  team  is  essential  before  the  onset  of  the 
test.  Interface  between  the  instrumentation  equipment  manufac¬ 
turers,  test  range  groups,  and  test  team  users  should  be  estab¬ 
lished  before  any  procedures  for  data  acquisition  and  reduction 
are  establisned. 

•  The  d*ca  manager  should  anticipate  physical  storage  space  and 
handling  restrictions  inherent  with  large  amounts  of  data. 

•  Transportation  needs  for  both  data  teams  and  data  from  the  col¬ 
lection  to  the  processing  point  should  be  thoroughly  addressed 
and  documented. 

•  Quick-look  analysis  for  individuals  other  than  the  test  team  must 
be  a  part  of  the  planning  process. 

•  Data  should  be  prioritized  in  terms  of  data  item  importance  and 
the  achievement  of  test  objectives.  This  is  important  if  time 
compression  occurs  (for  example,  earlier  than  expected  require¬ 
ments  for  data  analysis). 

7.  Instrumentation  Plan 

The  satisfactory  completion  of  any  test  program  requires  the 
availability  of  adequate  instrumentation  and  range  facilities.  The  appro¬ 
priate  location  for  the  planned  test  depends  on  the  type  of  instrumentation 
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required  which  is,  in  part,  determined  by  the  environment  desired  for  the 
test  (terrain,  weather,  etc.)  as  well  as  availability,  logistics,  economy, 
size,  and  other  factors.  The  unique  nature  of  joint  tests  often  requires 
specialized  instrumentation,  further  complicating  the  selection  of  appro¬ 
priate  range  facilities.  In  addition  to  affecting  the  selection  of  test 
facilities,-  the  availability  of  instrumentation  can  have  a  direct  impact  on 
the  ability  to  perform  required  measurements  as  specified  in  the  Test 
Design.  Early  planning  for  and  coordination  of  instrumentation  will  ensure 
the  availability  of  necessary  instrumentation  at  the  correct  time. 

The  size  and  complexity  of  the  task  for  the  JTD  can  be  strongly 
influenced  by  the  mission  and  capability  of  the  test  center  or  range 
selected.  Test  centers  (such  as  Eg! in  AFB  or  Naval  Weapons  Center,  China 
Lake)  have  ranges  which  are  normally  instrumented  to  gather  quantitative 
data  on  time,  space,  and  position  (TSPI)  and  on  system/subsystem  per¬ 
formance.  The  U.S.  Army  Combat  Developments  Experimentation  Command  (CDEC) 
is  a  test  center  with  instrumentation  and  data  processing  designed 
specifically  for  operational  tests.  It  has  a  strong  capability  for 
supporting  two-sided  tests.  On  the  other  hand,  training  ranges  (such  as 
Nellis  AFB  or  Ft.  Irwin)  have  very  little  instrumentation.  There  may  be 
training  equipment  (such  as  the  Air  Combat  Maneuvering  Instrumentation) 
which  provides  TSPI,  but  it  is  generally  limited  in  coverage  or  capability. 
Based  on  the  data  requirements  contained  in  the  Test  Design,  the  JTD  should 
review  with  the  Service  points  of  contact  the  availability  of  various  test 
center/range  facilities,  their  suitability  for  the  test,  and  the  instru¬ 
mentation  available  at  each  facility.  If  all  required  instrumentation  can 
be  made  available  at  a  specific  range  (or  ranges),  then  instrumentation 
planning  becomes  a  mere  scheduling  problem  to  ensure  that  the  range  and 
required  instrumentation  are  available  for  the  test.  The  planning  is 
complicated,  however,  if  specialized  instrumentation  is  required.  For 
test-unique  instrumentation,  the  JTD  should  identify: 

•  Nomenclature,  manufacturer,  and  model  number 

•  Number  required 
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•  Required  location 

•  Source 

•  Repair  and  cal ■ oration  responsibilities 

•  Date  and  required  duration  of  use 

•  Utilization  (e.g. ,  data  recording,  telemetry  package,  system 
checkout,  etc.) 

•  Procurement/development  schedule,  as  required 

•  Cost  estimate. 

Test  unique  instrumentation  is  the  responsibility  of  the  JTD.  If 
unique  requirements  exist,  the  JTD  and  his  staff  must  specify  the  equip¬ 
ment,  budget  for  it,  and  arrange  for  its  procurement. 

In  any  case,  the  JTD  should  submit  test  descriptions,  schedules, 
and  requirements  to  the  range  at  the  earliest  possible  date  to  ensure 
adequate  lead  time  for  integration  into  range  scheduling.  If  the  existing 
range  instrumentation  is  not  adequate,  the  JTD  should  coordinate  his  acqui¬ 
sition  of  test  unique  items  with  the  range  to  assume  that  they  can  be 
successfully  integrated.  It  is  possible  that  a  cooperative  instrumentation 
program  betwe  n  the  JTD  and  a  range  can  be  undertaken.  This  provides  the 
JTD  with  a  procuring  agency  and  is  a  natural  solution  to  the  problem  of 
materiel  disposition. 

8.  Logistics  Support  Plan 

Logistics  support  consists  primarily  of  services  and  facilities 
at  the  military  installation  where  the  test  is  being  conducted.  Many 
logistics  requirements  can  be  identified  and  a  determination  concerning 
their  availability  made  during  the  test  planning  phase.  The  JTD  should  pay 
particular  attention  to  defining  the  logistics  support  necessary  for  TDY 
personnel,  equipment,  vehicles,  and  aircraft  and  oversee  the  development  of 
a  Logistics  Support  Plan.  This  plan  may  be  separated  into  the  two  major 
areas  of  materiel  and  base  support  and  must  always  specify  both  the  dates 
needed  and  the  quantity  required, 
a.  Materiel 

This  section  of  the  plan  should  include  requirements  for 
supplies,  supply  accounts,  and  special  procedures.  Any  non-standard  items 
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which  the  installation  will  be  responsible  for  procuring,  stocking,  or 
handling  should  be  identified.  Requirements  for  POL,  gases,  and  other 
chemicals,  as  well  as  any  hazardous  handling  or  storage  aspects  must  be 
considered.  Munitions,  armament  maintenance,  and  explosive  ordnance  dis¬ 
posal  support  requirements  must  be  stated  along  with  requirements  for 
storage,  handling  and  delivery. 

b.  Base  Support 

Base  support  requirements  include  items  such  as  field  main¬ 
tenance,  organization  maintenance,  shop  services,  billeting,  food  service, 
transportation,  postal  services,  accounting  and  finance,  TDY  funding 
policies,  medical  support,  and  recreation  services.  Any  special  physical 
security  measures  or  courier  service  required  during  the  test  must  be 
identified.  Requirements  for  administrative  and  technical  facilities 
should  be  specified,  along  with  descriptions  of  necessary  floor  area,  door 
opening  dimensions,  lighting,  and  other  utility  requirements  if  they  are 
known. 

c.  lesson  Learned 

A  means  of  controlling  logistics  support  during  active 
testing  which  proved  effective  for  a  previous  JTF  was  the  establishment  of 
a  Joint  Logistics  Control  Center.  The  center  was  staffed  by  three  types  of 
logistics  personnel  (maintenance,  supply,  and  transportation);  it  provided 
centralized  control  and  kept  time  delays  in  locating  and  receivinq  needed 
parts  and  supplies  to  a  minimum. 

9.  Simulation  Plan 

Simulations  can  be  useful  in  many  phases  of  the  j'oint  testing 
process,  e.g. ,  from  early  preliminary  planning  to  analysis  of  data.  In  the 
information  gathering  phase,  a  simulation  can  be  extremely  useful  as  an 
instrument  to  learn  more  about  the  system  to  be  tested.  Various  aspects  of 
the  system's  response  to  different  stimuli  can  be  explored  as  well  as  the 
determination  of  the  degree  of  interaction  that  may  occur  when  several 
parameters  are  varied  simultaneously. 

In  the  categorization  of  variables  during  the  early  phase  of 
test  design,  a  helpful  estimate  of  the  importance  of  various  independent 
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variables  can  be  obtained  by  using  a  simulation  to  examine  the  sensitivity 
of  the  dependent  variables  to  changes  in  the  independent  variables.  In 
this  way  unimportant  independent  variables  can  be  dropped  from  considera¬ 
tion,  and  the  limited  resources  available  to  the  JTO  can  be  used  on  more 
interesting  and  productive  items.  The  simulation  can  also  be  used  to 
establish  the  range  of  interest  for  the  variables.  In  this  way  the  levels 
or  settings  for  the  primary  factors  can  be  ascertained  for  the  test  design. 

Once  the  preliminary  test  design  has  been  set,  the  test  condi¬ 
tions  can  be  "dry  run"  by  use  of  the  simulation  to  look  for  any  unexpected 
results  that  might  limit  the  scope,  range,  or  usefulness  of  the  test.  In 
certain  cases  (e.g. ,  IFFN)  the  simulation  might  be  the  main  instrument  of 
the  test  with  the  physical  tests  providing  inputs  to  the  simulation  and 
being  used  to  validate  the  model  at  several  specific  points.  The  validated 
simulation  can  then  be  run  to  provide  more  continuous  data  over  a  large 
number  of  test  conditions.  This  approach  should  not  be  attempted  without 
full  consideration  of  the  precautions  and  disadvantages  discussed  below. 

During  the  conduct  of  a  test,  simulations  can  serve  two  funda¬ 
mental  purposes.  First,  they  can  be  used  to  simulate  testable  events  as  a 
planning  tool  and  to  assist  in  the  evaluation  of  unanticipated  events. 
Second,  simulation  can  be  used  to  fill  untestable  gaps  (such  as  SAM  fly¬ 
outs).  The  latter  simulations  can  be  used  to  extend  the  evaluation  to 
condi tions/environments  beyond  those  provided  in  the  test.  Also,  this 
simulation  of  untestable  events  must  be  used  if  the  test  is  structured  for 
Real-Time  Casualty  Assessment.  However,  these  simulations  are  seldom  based 
on  empirical  data. 

Simulations  can  contribute  to  a  better  understanding  of  the 
capabilities  and  limitations  of  the  items  under  test.  They  provide  several 
advantages.  Simulations  can  be  accomplished  in  a  relatively  short  time-- 
given  the  existence  of  the  computer  program,  personnel  who  know  it,  and/or 
documentation  to  fully  explain  it.  The  cost  of  accomplishing  the  simula¬ 
tion  under  these  conditions  can  be  substantially  less  than  the  cost  of 
performing  comparable  physical  tests.  The  number  of  test  conditions  that 
can  be  explored  is  very  great.  Finally,  future  conditions  can  be  explored 


which  today  cannot  be  reproduced  physically.  For  example,  a  missile's 
performance  may  be  explored  against  a  future  enemy  target  whose  altitude 
and/or  speed  capability  cannot  be  physically  reproduced  within  the  present 
state  of  the  art. 

Simulations  are,  however,  not  without  disadvantages.  To  be 
successful  with  simulations  one  must  pay  close  attention  to  assumptions 
which  may  rule  out  the  use  of  the  simulation  for  the  purposes  desired. 
Frequently  the  assumptions  made  are  more  for  mathematical  convenience  than 
for  realism.  Simulations  are  most  useful  if  they  have  been  adequately 
validated.  In  most  cases,  the  range  of  validation  may  be  limited,  and  the 
information  in  previous  test  plans  and  test  reports  on  conditions  sur¬ 
rounding  the  data  acquisition  may  be  sparse.  If  large  models  must  be  used, 
the  key  to  control  is  continuity  of  analyst  personnel.  In  addition,  the 
simulation  should  be  user/analyst-oriented  (not  computer  programmer- 
oriented).  It  should  have  a  good  post-processor  to  tabulate  desired 
results  and  optimally  should  have  an  operationally-oriented  graphical 
output.  Finally,  appropriate  validated  simulations  are  not  readily  avail¬ 
able  for  most  JT&E  purposes.  The  JTD  should  contact  the  Service  and  JCS 
Studies  and  Analysis  agencies  for  expert  advice  on  simulations. 

Simulations  can  be  useful  in  many  phases  of  JT&E;  however, 
caution  is  advised  in  checking  on  inherent  assumptions,  as  well  as  on  the 
range  of  validation  the  simulation  has  attained.  If,  after  due  considera¬ 
tion  of  its  advantages  and  disadvantages,  the  decision  is  made  to  use 
simulations,  thei>  utilization  and  integration  into  the  test  must  be  care¬ 
fully  planned.  A  Simulation  Plan  must  be  prepared  to  address  the  two  broad 
areas  of  simulation  development  and  simulation  application.  Simulation 
development  can  be  separated  into  three  phases:  design,  programming,  and 
implementation.  The  design  phase  will  determine  how  and  what  to  simulate, 
data  requirements,  the  general  logical  structure  of  the  simulation,  and 
output  requirements.  The  programming  phase  will  include  the  coding, 
testing,  and  debugging  of  the  computer  programs.  Finally,  in  the  imple¬ 
mentation  phase,  the  simulation  will  be  installed  on  the  computer  on  which 
it  will  be  run. 


The  implementation  phase  can  be  lengthy  or  short,  depending  on 
the  nature  of  the  model (s)  to  be  used  in  support  of  the  test.  If  existing 
simulations  are  to  be  used,  the  design  and  programming  phases  are  elimi- 
nated,  leaving  only  the  requirement  to  implement  the  existing  model  on  a 
particular  machine.  However,  if  new  simulations  are  required,  the  develop¬ 
ment  aspect  of  simulation  support  could  be  a  critical  path  in  overall  test 
execution.  Even  relatively  straightforward  simulations  can  require 
considerable  time  to  design  and  program.  In  any  event,  the  JTD  should 
allow  sufficient  lead  time  to  have  the  simulation  support  available  when 
the  test  requires  it. 

Planning  for  the  application  of  the  simulations  is  especially 
critical  if  the  simulations  are  used  to  support  decision-making  for  sub¬ 
sequent  test  phases.  The  Simulation  Plan  should  specify  which  test  data 
will  be  required  for  input  to  the  simulation  and  should  provide  enough 
"turn-around*1  time  for  the  simulation  to  be  useful.  The  plan  should  also 
describe  any  "massaging"  of  test  data  required  to  put  the  data  in  a  form 
which  can  be  used  by  the  simulation.  This  aspect  of  the  simulation  plan 
should  also  be  reflected  in  the  Data  Management  Plan. 

C.  TEST  EXECUTION 

1 .  Program  Planning  and  Control 

The  process  of  managing  time  and  money  is  known  as  Program 
Planning  and  Control.  Many  tools  and  techniques  have  been  developed  to 
assist  managers  in  this  process.  One  of  the  most  complex  and  detailed  is 
the  DoO  Cost/Schedule  Control  System  (C/SCS)  which  establishes  very 
specific  methods  and  protocols  for  measuring  the  progress  of  large  pro¬ 
grams.  Other  much  less  complex  systems  are  in  widespread  use  throughout 
government  and  industry.  All  successful  program  control  systems,  whether 
complex  or  simple,  have  several  features  in  common.  These  are: 
e  Management  backi ng 

•  Formal  plan 
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•  Performance  measurement 

•  Periodic  reporting  of  deviations  from  plan. 

These  features  are  mandatory.  Lack  of  any  one  renders  the  system  useless 
as  a  management  tool. 

.  a.  Schedule  Control 

The  complex  nature  of  joint  tests  necessitates  a  highly 
structured  mechanism  for  program  planning  and  control.  Proper  program 
planning  and  control  provides  the  discipline  and  organizational  framework 
through  which  activities  ranging  from  test  planning  to  resource  commitment, 
instrumentation,  data  processing,  analysis,  and  reporting  can  be  coordi¬ 
nated.  With  proper  planning  and  control,  potential  pitfalls  can  be  iden¬ 
tified,  schedule  risks  highlighted,  and  protracted  delays  avoided. 

It  is  recommended  that  very  early  in  the  program  the  Joint 
Test  Director  select  an  individual  to  function  as  a  resource  manager.  The 
responsibilities  of  the  resource  manager  should  include  the  development  of 
a  master  program  plan  which  reflects  the  milestones  which  must  be  met  if 
the  test  is  to  begin  at  the  scheduled  time.  The  plan  should  be  capable  of 
illustrating  in  a  series  of  briefing  charts  the  overall  program  status  as 
well  as  progress  in  individual  areas  such  as  test  planning,  operational 
plan  development,  data  planning,  site  selection  and  preparation,  commitment 
of  personnel  and  materiel,  training,  instrumentation,  software  development, 
data  processing,  analysis,  and  reporting.  A  set  of  representative  program 
planning  and  control  briefing  charts  which  have  been  successfully  used  by  a 
Joint  Test  Force  may  be  found  in  Appendix  A. 

After  the  master  program  plan  with  its  milestones  has  been 
developed,  regular  program  status  review  briefings  should  be  conducted  to 
track  these  milestones.  These  briefings  serve  many  useful  functions. 
First,  they  keep  the  Joint  Test  Director  and  his  staff  well  informed  con¬ 
cerning  program  developments.  Second,  they  highlight  areas  requiring 
special  attention,  sometimes  in  the  form  of  "action"  items.  Third,  they 
provide  the  discipline  needed  to  keep  individual  staff  members  from  the 
natural  tendency  to  procrastinate  when  the  test  is  still  a  "long  way  off." 
Fourth,  they  readily  demonstrate  how  a  slip  in  one  part  of  the  schedule 
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impacts  on  other  areas.  Fifth,  they  provide  an  excellent  forum  for  the 
open  discussion  of  problem  areas. 

The  program  status  review  charts  are  useful  tools  for  his¬ 
torical  documentation.  It  is  recommended  that  the  charts  contain  not  only 
a  record  of  the  milestones  as  they  were  or  were  not  met  but  also  an  "events 
listing."  This  listing  should  contain  the  significant  briefings  and 
meetings  conducted  both  within  the  JTF  as  well  as  with  external  organiza¬ 
tions.  Such  a  listing  is  useful  as  a  reference  for  the  current  JTF.  At 
the  conclusion  of  a  test,  copies  of  the  program  status  review  charts  should 
be  made  a  part  of  the  JT&E  library  maintained  by  the  OOTE.  The  charts  can 
then  serve  the  function  of  "roadmaps"  for  future  Joint  Test  Forces, 
b.  Financial  Planning  and  Control 

(1 )  Background 

The  JTD  is  responsible  for  the  management  and  super¬ 
vision  of  all  aspects  of  the  JT&E.  One  very  important  aspect  is  the 
establishment  and  maintenance  of  a  sound  financial  management  system.  The 
JTD  must  control  and  account  for  all  OSD  funds  and  monitor  Service  expenses 
related  to  the  JT&E.  At  the  same  time,  funding  requirements  must  be 
developed,  maintained,  updated,  and  submitted  to  the  DOTE  and  the  Services, 
as  appropriate. 

(2)  Funding  Responsibilities 

Early  in  the  seventies,  Congress  supported  the  Blue 
Ribbon  Committee  findings  by  establishing  an  appropriation  for  JT&E.  OSD 
subsequently  established  a  program  element  (65804D)  which  funds  the  unique 
costs  of  directing,  planning,  and  executing  JT&E.  In  addition  to  funding 
JT&E,  this  program  element  supports  improvement  efforts  at  major  ranges  and 
test  facilities,  as  well  as  independent  tests  and  evaluations  ’irected  by 
Congress  or  the  SECDEF. 

All  current,  completed,  or  cancelled  JT&E  have  been 
funded  from  this  program  element.  In  addition  to  the  JT&E-unique  costs 
funded  from  this  program  element,  there  are  Service  O&M  costs  incurred 
for  items  such  as  personnel,  operation,  and  maintenance.  Because  of  the 
dual  funding  of  joint  tests,  the  budgeting  and  programming  process  is  a 
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cooperative  effort  which  involves  OSD,  the  Lead  Service,  and  all  other  Ser¬ 
vices  which  are  participating  in  a  joint  test.  The  DoD  Budget  Guidance 
Manual  (DoO  71 10-1 -M)  states: 

Funding  for  DDTE-di rected  JT&E  is  under  the  Director  of 
Test  and  Evaluation,  Defense  appropriation  (program 
element  65804D).  The  costs  incurred  under  this  program 
element  are  for  the  direction,  supervision,  and  per¬ 
formance  of  JT&E  and  will  be  for  those  areas  which  are 
unique  to  the  needs  of  the  JT&E.  In  the  accomplishment 
of  DDTE-di rected  joint  tests,  the  Components  will  be 
reimbursed  from  these  RDT&E  funds  for  any  unique  costs 
associated  with  the  directed  tests  with  the  exceptions 
of  normal  military  pay,  0&M  expenditures,  and  the 
hardware  utilized  for  the  test.  When  directed  as 
Executive  Agent  for  a  particular  JT&E,  the  Component 
will  be  responsible  to  the  Secretary  of  Defense  for 
ensuring  that  all  resources  necessary  for  the  success¬ 
ful  accomplishment  of  the  JT&E  are  available  to  the 
Joint  Test  Director  (JTD).  This  includes  administra¬ 
tive  management  support  and  facilities  for  the  JTD. 

The  Components1  support  costs  (O&M,  Procurement,  and 
RDT&E)  are  to  be  programed  and  budgeted  for  in  accor¬ 
dance  with  their  established  budgetary  procedures. 

Thus,  costs  which  are  incurred  solely  as  a  result  of 
conducting  a  JT&E  and  which  cannot  be  regarded  as  providing  long-term 
mission-oriented  benefits  (such  as  training  or  readiness)  are  normally 
funded  by  OSD  in  Program  Element  65804D.  However,  the  precise  line  between 
costs  which  are  the  responsibility  of  one  of  the  participating  Services  and 
costs  which  should  be  funded  or  reimbursed  by  OSO  is  not  always  easy  to 
define.  In  many  circumstances,  the  propriety  of  the  fund  source  may  have 
to  be  decided  by  a  Comptroller  opinion.  Nevertheless,  the  guidelines  pro¬ 
vided  in  Table  III-l  provide  a  sound  basis  for  establishing  budget 
requirements. 

(3)  Financial  Management  System 
(a)  Introduction 

As  previously  noted,  an  experienced  financial 
manager  is  an  essential  element  of  the  JTD  staff.  He  is  responsible  to  the 
JTO  for  establishing  and  maintaining  a  complete  financial  program  with  an 
updating/tracking  system.  The  remainder  of  this  section  of  the  manual  is 
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TABLE  III-l.  JT1E  FUNDING  RESPONSIBILITIES 


E 


j 


i 


FUNDED  BY 


CATEGORY  COST 


LEAD  SERVICE 
B  PARTICIPATING 
OSD  SERVICES 


1.  JTOANO  STAFF 


A. 

TRAVEL 

X 

B. 

PER  DIEM 

X 

C. 

NEW  CIVILIAN  HIRE 

X 

0. 

WORO  PROCESSOR 
(RENTAL  OR  PURCHASE) 

X 

(11 

c. 

OPTICS  EQUIPMENT 

X 

p. 

SUPPUES 

X 

Q. 

OPPICE  FACILITIES 

X 

H. 

(BUILD.  LEASE.  ALTER) 

VEHICLE  RENTAL/ LEASE 

X 

1. 

TECHNICAL  AOVISQRS 

X 

2.  CONTRACTOR 

A.  OESIQN  DEFINITION.  TEST 

DESIGN.  INDEPENDENT  ANALYSIS 

A  TECHNICAL  SUPPORT  X 

S.  SIMULATION  X 


3.  TRAINING 


A. 

TEST  ROE  ANO  OATA  COLLECTION  TRAINING 

FOR  WHITE  CONTROLLERS 

X 

m 

B. 

TEST  UNIQUE  C>  DOCTRINE  ANO  TACTICS 

TRAINING  FOR  RED  PLAYERS 

X 

HI 

C. 

TEST  UNIQUE  TACTICS  TRAINING  FOR  BLUE  PLAYERS 

X 

0 

OTHER  TRAINING— TECHNICAL— JTO  STAFF 

X 

HI 

4  FACILITIES 

A.  BUILDINGS  -  CONSTRUCTION. 
LEASE.  ALTERATION 

a.  TRAILERS  -  LEASE  ANO  PURCHASE 
C.  HAROWARE/AOP 
O.  RANGE  COSTS 
E.  INSTRUMENTATION 
F  SURROGATES 

G.  STE  PREPARATION 

H.  BASE  OPERATING  SUPPORT 

I.  PLATER  COMM  SUPPORT 

J.  UTUTtES  COST 

X.  OFF  SITE  LEASING 

L.  FLAYER  VEHICLE  LEASING 

M.  PHOTO/VIDEO  SUPPORT 

N.  OPSEC  SURVEY 

0  ENVIRONMENTAL  IMPACT  STUOY 
P  PLAYER  SUPPLY  ACCOUNTS 


! 

* 

l 


X 

X 

X 

X 

X 

X 

X 


X 


X(B) 

m 


X 

X 

X 


i 


X 

NEGOTIABLE 

X 


X 


X 


TABLE  III-l.  JT&E  FUNDING  RESPONSIBILITIES  (CONTINUED) 


PUHOEO  BY 

LEAO  SERVICE 
A  PARTICIPATING 


CATEGORY 

COST 

OSO 

SERVICES 

TEST  SUPPORT 

A. 

REO  AND  BLUE  FORCE  TRAVEL  AND  PER  DIEM 

X 

a. 

WHITE  FORCE  TRAVEL  ANO  PER  DIEM 

X 

c. 

TEST  VEHICLE  MODIFICATION. 

OEMOOIFICATION  &  TRANSPORTATION 

X 

D. 

SURROGATE  MODIFICATION.  OEMOOIFICATION 

A  TRANSPORTATION 

X 

1 

FLYING  HOURS 

X 

F. 

VEHICLE  HOURS 

X(4| 

G. 

FUEL 

X 

H. 

PARTS  (TEST  VEHICLES)  AND 

TRANSPORTATION 

X 

1. 

MUNITIONS 

X 

J. 

TRANSPORTATION  OF  MUNITIONS 

X 

K. 

OATA  COLLECTION.  ANALYSIS  ANO  STORAGE 

X 

L 

EQUIPMENT  LOSSES 

X 

M. 

PROPERTY  DAMAGE 

X 

N. 

INDUSTRIALLY  FUNDEO  FUGHT  HOURS 

X 

0. 

SIMULATION  COSTS.  MODIFICATION. 

OEMOOIFICATION.  ANO  TRANSPORTATION 

X 

(3) 

MOW 

(II  tX'STWMl  AMO  SUtTAMJ  SfltVICt  MSOURCII.  PACILITIIt,  ITC ..  WHICH  CAM  M  MAOS  AVARAMJI  WHjL 1*  PHOVtOSO:  HOWtVtN.  AOOITlOMAA 
fttSOUHCtt.  COSTS.  ITC  .  WIU  SI  niMOlO  ST  OSO 

(S)  DUSTIMO  AMO  SUITASU  SSSVICI  AAMO«S/TI»T  VACfUTIIS  WHICH  CAM  M  MAOS  AVAILASU  WIU  St  PHOVIOtO:  MOWSVtH.  AMT 
SSMMSUSSSMSNT  COSTS  HSU  St  HjMOtD  SV  OSO 

I*  intTIMO  AMO  SWfTASU  IfSVICt  IlMULATTOM  CAfASlUTIta  WHICH  CAM  St  MAOt  AVAUASU  WIU  SI  PHOVIOtO.  HOWtVtS.  OTHIN 
AOOmOMAi  COSTS  rMCLUOtMO  MOOlWCATtON.  THAMSPOSTATH>N  ANO  MAIMTtNAMCf  WIU  SI  FUNOtD  SV  OSO- 
|4|  IN  CAStS  OS  CXTHtMt  AMO  UNUSUAL  0«0« AOATiOM  OS  VfHICLt  US1.  THt  StSULTAMT  SUNOIHO  OS  THlSI  COSTS  WIU  St  MtOOTIATIO 
(S)  CXACT  SASTmON  os  SUMOIMO  SOS  CONSTRUCTION.  LIAS  I  ON  ALTS  NATION  IS  OtStNOCNT  ON  WHtTHSN  THt  WORK  *  SStQSICAUV  SON 

rnc^rst. 
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not  intended  to  be  a  detailed  tutorial,  but  rather  a  brief  description  of  a 
successful  financial  program  content  system.  Any  system  will  fail  if  it  is 
not  methodically  and  rigorously  enforced.  Although  particular  tests  may 
differ  in  detail  from  the  standard  described,  the  general  principles  out¬ 
lined  are  sound  and  must  be  followed. 

The  JTD  should  note  from  the  outset  that,  although  he 
may  control  the  0S0  funds  and  track  and  account  for  Service  expenditures, 
he  will  be  dependent  upon  supporting  finance  offices  for  their  actual 
administration.  This  is  analogous  to  a  unit  commander  or  program  manager 
who,  while  controlling  commitments  and  obligations,  depends  on  his  instal¬ 
lation  finance  office  to  disburse  the  funds.  What  is  not  analogous  is  that 
the  JTD  is  a  temporary  entity  without  long-standing  ties  to  a  finance 
office.  Therefore,  the  JTD  and  his  staff  must  establish  new  and  effective 
relationships  for  each  JT&E. 

A  fictional  JT&E,  "ABCVAL"  will  be  used  to  illustrate 
the  implementation  of  a  representative  JT&E  financial  program.  The 
scenario  for  ABCVAL  is  summarized  in  Figure  I I 1-2. 

(b)  Financial  Program 

The  JTD  and  his  staff  will  create  and  update, 
whenever  appropriate,  a  complete  financial  program.  The  main  breakdowns  in 
the  financial  program  will  be  as  follows: 

(1)  Appropriation 

(a)  ROT&E 

(b)  O&M 

(c)  Procurement 

(2)  Program  Element  (P. E. ) 

(a)  658040 

(b)  XXXXXF 

(c)  XXXXXA  Service  P.E.s  as  required  and  authorized 

(d)  XXXXXN  by  the  Services 

(3)  Fiscal  Year 

(4)  Functional  Tasks. 
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"ABCVAL" 


ABCVAL  -  AN  OSD-OIRECTED  JOINT  TEST  AND  EVALUATION 

INVOLVING  ARMY,  AIR  FORCE,  AND  MARINE  FORCES. 
NOMINATED  IN  1979 
BUDGET  BOGIE  FROM  NOMINATION  $15M 
FEASIBILITY  STUDY  COMPLETED  JUST  AFTER 
JAN  80  PRESIDENT'S  BUDGET  SUBMISSION 
FEASIBILITY  STUDY  SHOWS  A  REFINED  RESOURCE 
ESTIMATE  OF  S20M 

FINANCIAL  PLAN  SHOWS  JAN  80,  JUL  80, 

NOV  80  AND  JAN  81  BUDGETS 
"TIME"  IS  NOW  LATE  1981  -  A  NEW  BUDGET 
UPDATE  IS  BEING  PREPARED,  NOT  YET  COMPLETE 
—  SMALL  GROWTHS  HAVE  OCCURRED  IN  SOME 
ELEMENTS  BUT  ARE  OFFSET  BY  REDUCTIONS 
IN  OTHER  ELEMENTS. 

OBLIGATIONS  FOR  FY  82  FUNDS  HAVE  BEGUN. 


Figure  III-2.  A  Scenario  for  Financial  Planning  Example 
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In  the  large  majority  of  joint  tests,  the  only 
funds  directly  controlled  by  the  JTD  are  those  provided  by  the  DDTE  out  of 
P.E.  658040.  The  following  section  uses  658040  RDT&E  funds  in  the  sample 
charts.  If  other  funds  are  placed  under  the  direct  control  of  the  JTD,  the 
same  techniques  are  directly  applicable  and  require  only  that  separate 

tracks  be  maintained  for  each  source  of  funds.  The  essential  elements  of 

financial  planning  and  control  are  listed  in  Figure  I I 1-3.  A  brief  summary 
of  each  element  follows.  The  numbers  cited  are  keyed  to  those  in  the 
figure. 

1.0  Current  Financial  Program;  The  totality  of  all  management  actions 
required  to  request,  control,  and  account  for  funds. 

1.1  Financial  Tracking  Plan:  Provides  a  current  top-level  view  of  funds 
required  and  funds  available.  It  also  provides  a  track  from  the  first 
budget  submission  through  each  change  to  the  present.  It  is  updated 
with  each  budget  submission  or  approved  program. 

1.2  Program  Budget:  The  latest  submission  of  the  budget  for  the  Program 

Element/Fund  Category.  If  requirements  significantly  exceed  the 

former  budget  submission,  an  update  should  be  made.  When  replaced  by 

a  new  update,  the  old  program  budget  becomes  the  latest  entry  in 
section  2.0.  It  is  updated  with  each  budget  submission  or  approved 
program. 

1.3  Total  Obligations:  Provides  a  breakout  of  funds  obligated  by  func¬ 
tional  task  and  by  fiscal  year.  It  is  updated  as  obligations  occur. 

1.4  Current  Fiscal  Year  Spending  Plan:  Covers  two  fiscal  years  for  RDT&E 
funds,  three  for  procurement,  and  one  for  0&M.  This  plan  is  more 
detailed  than  the  budget.  Each  entry  is  a  distinct  type  of  end- item 
or  service.  It  is  updated  monthly  or  more  often  as  required. 

2.0  JTF  Budgets:  A  track  leading  from  the  first  budget  submission  to  the 
present.  These  entries  consist  of  the  Program  Budgets  (Item  1.2) 
after  they  have  been  superseded.  The  list  of  JTF  budgets  mirrors  each 
entry  in  the  Financial  Plan  Track  but  in  greater  detail.  Each  budget 
should  be  accompanied  with  explanatory  notes  regarding  the  budget 
changes. 
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FINANCIAL  PROGRAM 


1.0  CURRENT  FINANCIAL  PROGRAM 

1 . 1  FINANCIAL  TRACKING  PLAN 

1.2  PROGRAM  8U0GET 

1 .3  TOTAL  OBLIGATIONS.  BY  FISCAL  YEAR 

1 .4  CURRENT  FISCAL  YEAR  SPENDING  PLAN 
2.0  JTF  BUDGETS 

2.1  7/80 

2.2  11/80 

2.3  (AS  REQUIRED  FOR  FUTURE  UPDATES) 

2.4 


3.0  FUND  RELEASES  FOR  EACH  FISCAL  YEAR 

3.1  FY  81 

3.2  FY  82 

3.3  FY  83 

3.4  FY  84 

3.5  (AS  REQUIREO  TO  COMPLETE  PROGRAM) 

36 

• 

• 

4.0  OBLIGATIONS  FOR  EACH  FISCAL  YEAR 

4.1  FY  81 

4.2  FY  82 

4.3  FY  S3  ' 

4.4  FY  84 

4.5  (AS  REQUIRED  TO  COMPLETE  PROGRAM) 

4.8 

• 

• 

5.0  EXPENDITURES  AND  DISBURSEMENTS  FOR  EACH  FISCAL  YEAR 

5.1  FY  81 

5.2  FY  82 

5.3  FY  83 

5.4  FY  84 

5.5  (AS  REQUIRED  TO  COMPLETE  PROGRAM) 

5.6 


Figure  II 1-3.  Essential  Elements  of  Financial  Planning  and  Control 
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3.0  Fund  Releases:  Records  of  funds  released  by  fiscal  year,  by  fund 
category,  and  by  program  element.  The  record  is  organized  Dy  release 
date  and  includes  the  allotment  (or  Military  Interdepartmental  Pur¬ 
chase  Request--MIPR)  identification,  the  receiving  finance  office,  the 
end-i tems/services  for  which  the  money  was  alloted,  and  the  amount. 
They  are  updated  as  fund  releases  occur. 

4.0  Obligations:  A  track  by  fund  category  and  program  element  of  obliga¬ 
tions  in  a  fiscal  year.  This  record  is  kept  by  contractual  instrument 
since  a  contract  or  a  supplemental  agreement  is  the  method  of  obliga¬ 
tion.  It  is  updated  as  obligations  occur. 

5.0  Expenditures  and  Disbursements:  A  record  of  the  funds  expended  by  and 
funds  paid  to  individual  contractors.  It  is  updated  monthly. 

A  more  detailed  discussion  of  the  individual  elements  of  the 
financial  plan  along  with  suggested  formats  for  recording  the  relevant 
information  follows. 

L  The  Financial  Tracking  Plan 

It  is  essential  for*  the  JTD  and  his  staff  to 
create  and  update,  whenever  appropriate,  a  Financial  Tracking  Plan.  At  the 
end  of  a  test,  the  initial  Financial  Tracking  Plan  and  its  subsequent 
versions  (tracks)  should  be  a  complete,  top-level,  financial  history  of  the 
test  funds  under  the  control  of  the  JTD.  It  will,  of  course,  include 
65804D  funds,  but  should  also  provide  a  record  of  any  Service  funds  placed 
under  the  management  authority  of  the  JTD.  Figure  I I 1-4  shows  the  progress 
of  the  Financial  Tracking  Plan  from  the  first  "bogie"  through  the  update 
resulting  from  the  feasibility  study,  and  a  better,  more  realistic  estimate 
after  detailed  test  planning  was  begun.  The  JTD  should  ensure  tnat  his 
resource  manager  has  an  accounting  of  why  significant  changes  in  the  plan 
have  occurred. 

2.  Current  Program  Budget 

Having  constructed  the  top-level  budget 
estimate  for  submission  to  OSD  and  inclusion  in  the  Financial  Tracking 
Plan,  the  next  management  tool  to  consider  is  the  current  budget  for  the 
entire  test,  broken  out  to  functional  task  level.  Figure  I I I - 5  displays 
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CURRENT  PROGRAM  BUDGET 


Figure  1 1 1-5.  Current  Program  Budget 


the  current  program  budget.  Again,  budgets  must  be  prepared  for  each  fund 
category  and  program  element.  Each  budget  submission  reflected  in  the 
Financial  Tracking  Plan  should  have  a  complete  program  budget  prepared. 
Further,  each  functional  task  entry  should  be  tracked  from  the  previous 
budget.  The  program  budget  provides  an  overview  of  the  structure  of  the 
program  by  major  functional  tasks  and  provides  a  summary  of  how  the  funds 
are  spread  over  the  length  of  the  program.  Review  of  the  budget  provides 
feedback  as  to  the  consistency  of  the  test  plan.  Some  attributes  are 
obvious.  The  threat  simulators/surrogate  vehicles  must  be  funded  early  on 
(front-end  loaded)  to  ensure  their  availability  for  the  test.  Some  of  the 
other  elements  (such  as  data  management)  peak  much  later,  but  still  demand 
early  commitment  of  funds  to  guarantee  that  the  full  capability  will  be 
available  for  the  actual  test. 

3.  Total  Obligations-to-Oate 

Another  long-term  financial  document  is  the 
Total  Obligations-to-Oate.  (See  Figure  III-6.)  In  content  and  in  form, 
the  Total  Obligations-to-Oate  documents  are  parallel  to  the  Program  Budget 
Form.  These  charts  represent  the  status  of  obligated  dollars,  that  is, 
dollars  which  have  been  legally  obligated  by  completing  an  appropriate 
contractual  instrument  or  by  a  fund  transfer  to  a  different  department  of 
the  Government.  Obligation  of  dollars  is  a  most  important  milestone  since 
Defense- appropriated  dollars  have  fixed  lifetimes.  If  the  dollar  lifetime 
(for  example,  two  years  on  RDT&E  money)  expires  before  obligation  occurs, 
the  money  is  returned  to  the  U.S.  treasury  and  is  no  longer  available. 
Thus,  for  an  RDT&E  account  as  is  normally  the  case  for  JT&E  unique  funds, 
the  obligations  must  be  complete  in  two  fiscal  years  (as  shown  in  the 
figure).  In  other  words,  only  the  two  most  recent  fiscal  years  should  show 
obligations  less  than  the  approved  program  for  that  particular  year. 
Actual  expenditures  (e.g. ,  payments  to  contractors)  can  extend  beyond  the 
two-year  limit.  Further,  contracts  can  be  incrementally  funded,  that  is, 
the  cognizant  finance  office  can  transfer  funds  into  the  contract  at 
prescribed  intervals  during  the  course  of  the  programs.  Incremental 
funding  does  not  affect  the  obligation  unless  money  from  more  than  one 
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fiscal  year  is  involved.  In  that  case,  the  obligation  may  be  for  the  total 
amount  including  future  fiscal  years,  but  the  obligation  for  the  future 
years  does  not  take  force  until  the  Congress  appropriates  that  fiscal 
year's  money.  The  Obligations-to-Oate  cha-t  should  be  updated  at  every 
budget  update. 

4.  Current  Spending  Plan 

The  current  spending  plan  (Figure  I I I- 7)  is  a 
shorter  term  management  tool  and  should  be  updated  monthly.  As  long  as  the 
total  requirements  remain  within  the  bounds  of  the  currently  approved 
budget,  no  external  update  is  required.  However,  if  the  spending  plan 
indicates  that  the  current  budget  will  be  exceeded  in  any  fiscal  year  or  a 
significant  underrun  will  occur,  the  JTD  should  provide  an  updated  budget 
to  the  DOTE.  Again,  for  the  RDT&E  appropriation,  any  one  spending  plan 
will  encompass  the  two  fiscal  years  for  which  money  is  currently  valid. 
The  agency  column  contains  the  name  of  an  agency  through  which  the  funds 
will  be  obligated.  Items  in  this  breakdown  are  finer  grained  than  the 
budget  or  obligations  breakdown.  The  committed  column  refers  to  funds 
which  the  JTF  has  directed  the  appropriate  comptroller  or  finance  officer 
to  reserve  for  a  given  contractual  effort.  Commitment  does  not  imply  any 
legal  requirement  for  government  expenditure.  Obligation  occurs  when  a 
legal,  binding  contractual  instrument  is  executed.  The  (Est)  Actual  date 
column  in  Figure  II 1-7  is  an  estimated  obligation  date  for  committed  funds 
and  actual  date  for  obligated  funds.  It  is  important  to  coordinate  with 
the  procuring  agency  on  obligation  dates  to  verify  that  the  schedule 
envisioned  is  achievable. 

The  final  elements  of  the  current  spending 
plan  are  the  running  totals  of:  Approved  Program  Funds,  Released  Funds, 
Obligated  Funds,  Committed  Funds,  and  Uncommitted  Funds.  For  RDT&E  funds, 
good  financial  practice  calls  for  all  funds  to  be  obligated  during  the 
fiscal  year  in  which  they  are  appropriated.  However,  the  money  is  good  for 
two  years  and  small  deviations  from  the  goal  can  be  tolerated.  Procurement 
funds  have  a  different  obligation  schedule  and  O&M  as  one  year  money  must 
be  obligated  in  the  year  appropriated. 
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CURRENT  SPENDING  PLAN— AS  OF  12/15/81 


ITEM 

AGENCY 

COMMITTED 
(*  IN  1000*1 

OBLIGATED 
(»  IN  1000«| 

(EST) 

ACTUAL  DATE  OF 
OBLIGATION 

FY  81 

TEST  DESIGN 

IOA 

OSO 

SO 

10/1/80 

JTF/TVL/TRANS 

BASE  "A" 

20 

9/30/81 

FY  82 

TEST  DESIGN 

IOA 

OSO 

150 

10/1/81 

ANALYSIS 

COMPANY  "X" 

APCMD 

100 

10/ IB/81 

OATA  MANAGEMENT 

COMPANY  "Y" 

AFCMO 

160 

(2/2B/82) 

INSTRUMENTATION 

RANGE  "8" 

FORT  -B" 

100 

0 

10/15/81 

COMPANY  "Z“ 

FORT  -B" 

100 

(3/1/82) 

THREAT  SURROGATES 

SAM  -YY" 

.  AF/ASO 

1630 

10/30/81 

TANK  ZZ - 

USA/TAR  AOCOM 

800 

10/30/81 

JAMMER  -QQ" 

NAVELEX 

200 

(12/20/81) 

JTP/TVL/ TRANS 

BASE  "A" 

129 

(9/30/82) 

SUMMARY 
(•  IN  THOUSANDS) 


FY  81 

PROGRAM 

70 

RELEASEO 

70 

08LIGATE0 

70 

COMMITTED 

0 

UNCOMMITTED 

0 

FY  82 

PROGRAM 

3419 

RELEASEO 

2909 

OBLIGATED 

2330 

COMMITTED 

579 

UNCOMMITTED 

510 

Figure  I I 1-7.  Current  Spending  Plan  -  As  of  12/15/81 
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5.  JTF  Budget  Estimate 

The  budget  estimate  is  the  JTD's  tool  for 
formally  communicating  his  fund  requirements  to  the  DOTE.  Again  the  budget 
estimate  must  be  broken  down  by  fund  category,  program  element,  fiscal 
year,  and  functional  task.  The  JTD  should  be  wary  of  a  too  detailed  sub¬ 
mission  to  higher  echelons.  Unnecessary  detail  destroys  the  clarity  of  the 
budget  submission  and  reduces  the  flexibility  of  the  JTD  and  of  the  DOTE. 
The  budget  estimate  is  a  crucial  document  for  two  reasons: 

•  It  is  the  basic  document  which  underlies  the  entire  financial 
structure  of  the  test. 

•  It  is  a  clear  message  to  the  DDTE  and  OSD  concerning  the  finan¬ 
cial  well-being  of  the  test. 

A  sample  budget  estimate  is  shown  in  Figure  I I 1-8.  The  financial  documen¬ 
tation  should  contain  a  copy  of  each  distinct  budget  estimate  submitted. 
As  indicated  in  the  Financial  Plan  (Figure  III-4),  a  budget  estimate  will 
be  permanently  retained  for  the  submissions  of  July  and  November  1980,  with 
the  latter  annotated  to  indicate  that  the  approved  President's  Budget 
submission  left  it  unchanged. 

6.  Fund  Releases 

Another  dimension  to  the  problem  of  financial 
management  in  the  JTF  is  that  money  must  be  released/allocated  to  the 
finance  office  which  supports  the  procuring  agency.  Thus,  funds  must  be 
tracked  by  Military  Interdepartmental  Purchase  Request  (MIPR)  or  allot¬ 
ments.  Sample  forms  for  FY81  and  FY82  are  shown  in  Figures  III-9  and 

I II- 10,  respectively.  Note  that  the  level  of  detail  in  these  forms  is 
essentially  the  same  as  the  spending  plan.  These  documents  should  be 
updated  whenever  a  fund  release  occurs. 

7.  Obligations  by  Fiscal  Year 

For  each  fiscal  year  in  which  funds  are 

expended,  a  updated  record  of  obligations  must  be  maintained. 

Figure  III-ll  provides  an  example  for  FY82. 
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Figure  III-10.  FY82  Fund  Releases 
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8.  Expenditures  and  Disbursements  by  Fiscal  Year 
There  are  two  aspects  of  expenditures  which 

the  JTD  must  track.  First,  he  must  know  the  rate  at  which  his  contractors 
are  expending  their  contract  dollars.  This  information  should  be  provided 
on  a  monthly  basis  by  the  contractors  directly  to  the  JTD.  Second,  he  must 
track  the  actual  disbursement  of  funds  from  the  Government  office  respon¬ 
sible  for  paying  vouchers  submitted  by  contractors.  This  information  must 
be  requested  on  a  continuing  basis  since  the  paying  office  usually  does  not 
provide  it  automatically.  This  data  will  serve  primarily  as  historical 
information  since  it  is  likely  to  be  3  to  6  months  before  disbursement 
information  is  available. 

A  sample  format  for  tracking  expenditures  and 
disbursements  is  contained  in  Figure  III-12.  Note  that  the  two  categories 
are  tracked  on  a  single  form  so  that  an  audit  trail  can  be  established 
linking  obligations,  expenditures,  and  disbursements. 

9.  Financial  Reporting  to  the  DOTE  . 

Program  Element  65804D  funds  are  appropriated 
annually  and  available  for  obligation  during  the  year  of  appropriation  and 
the  following  year.  Financial  managr  sic  criteria  for  the  overall  DDTE 
program  include  the  amount  of  appropriated  funds  which  are  obligated  and 
expended  by  the  end  of  the  first  year.  To  better  align  the  execution  of  PE 
658040  programs  with  the  appropriation  process,  the  DDTE  has  established 
the  following  reporting  procedures  which  must  be  followed: 

a.  PE  658040  Expenditure  Requests 

Each  request  to  the  DDTE  for  authori¬ 
zation  to  expend  funds  from  PE  65804D  (by  MIPR,  allotment,  or  request  for 
contract  services)  shall  comply  with  incremental  funding  procedures  and 
include  the  following: 

•  Identification  of  materiel  or  services  to  be  procured  or 
other  items  to  be  supported  by  PE  65804D  in  accordance  with 
the  appropriation  language  and  the  DoO  Sudget  Guidance 
Manual 

•  Justification  of  the  need  for  such  materiel,  services,  and 
other  items 
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•  Expected  date  of  contract,  contract  amendment,  or  other 
obligating  action 

•  Expected  expenditure  by  quarter  against  the  obligation. 

b.  Quarterly  Report  of  Obligations 
and  Billings 

Each  Joint  Test  Director  or  other  user 
of  PE  65804D  funds  shall  submit  to  the  DDTE  quarterly  a  data  sheet  for  each 
MIPR  or  allotment  and  for  each  contract.  For  contracts  sponsored  within 
ODDTE,  the  cognizant  staff  officer  shall  maintain  similar  data.  The  format 
to  be  used  when  submitting  these  quarterly  reports  is  depicted  in 
Figure  III-13. 

10.  Summary 

The  Financial  Plan  and  its  component  forms 
provide  the  JTD  with  a  complete  set  of  basic  tools.  The  precise  implemen¬ 
tation  may  be  varied  to  suit  the  size  and  complexity  of  the  specific  JT&E. 
However,  the  principles  of  constant  review,  updating  when  changes  occur, 
and  stringent  tracking  of  transactions  must  be  conscientiously  applied. 
The  resource  management  system  must  be  oriented  toward  early  identification 
and  resolution  of  financial  problems. 

2.  Test  Conduct  and  Control 

Joint  tests,  by  their  very  nature,  involve  multidimensional 
functions  and  exploring  the  unknown  or  uncertain.  Consequently,  they 
always  carry  a  degree  of  risk.  The  JTD  should  be  prepared  for  this  risk 
and  attempt  to  reduce  it  with  thorough  planning  and  attention  to  detail. 
A  test  of  only  moderate  complexity  normally  requires  that  the  efforts  of 
many  individuals  and  items  of  equipment  with  different  jobs  and  capabili¬ 
ties  be  combined  to  achieve  a  common  goal:  obtaining  valid  information  in 
a  form  which  permits  analysis  and  the  drawing  of  scientifically  sound 
conclusions. 

The  demands  imposed  by  the  testing  environment  will  require  the 
JTD  to  make  numerous  technical  and  operational  decisions.  These  decisions 
are  ascendant  over  routine  administrative  and  operational  matters  and  must 
be  weighed  in  relation  to  the  impact  on  test  design,  test  objectives  and 
quality  and  validity  of  test  data.  Since  joint  tests  vary  widely  in  scope. 
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complexity,  and  urgency,  it  is  not  possible  to  describe  precisely  how  the 
JTD  should  go  about  day-to-day  test  direction.  There  are,  however,  several 
precepts  which  have  general  application  to  all  joint  tests,  and  these  are 
discussed  below. 

a.  JTF  Indoctrination 

The  JTD  must  indoctrinate  members  of  the  JTF  concerning  the 
background  of  the  test  program,  test  items,  and  how  the  test  will  be  con¬ 
ducted.  The  indoctrination  can  be  either  on  a  group  or  individual  basis, 
depending  on  circumstances.  In  either  case,  JTF  members  must  be  thoroughly 
familiar  with: 

e  The  concept  of  operations,  including  the  way  available  resources 
will  be  used  to  meet  test  objectives 

•  Data  requirements,  and  how  data  will  be  collected,  processed, 
analyzed,  evaluated,  and  recorded 

e  Milestones  and  target  dates  for  all  phases  of  the  test 

e  Responsibilities  of  each  support  activity  and  how  its  efforts 
will  affect  the  total  test  effort 

•  Individual  responsibilities,  and  how  they  relate  to  one  another, 
including  responsibilities  for  preparing  portions  of  the  draft 
test  report  during  and  after  testing 

•  The  chain  of  authority  for  purposes  of  test  conduct  and  pro¬ 
cedures  for  interaction  and  coordination  with  other  agencies, 
organizations,  and  commands. 

b.  Test  Support  Verification 

Prior  planning  and  coordination  will  have  identified  test 
support  requirements.  Once  the  JTF  is  situated  at  the  test  site,  the  JTD 
should  verify  the  support  each  agency  has  been  asked  to  provide  and,  if 
possible,  meet  with  the  head  of  that  agency.  He  should  ensure  that  all 
actions  required  to  obtain  that  support  have  been  accomplished  prior  to  the 
scheduled  start  date  of  active  testing.  Problems  with  test  support  should 
be  identified  and  resolved  as  early  as  possible. 
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c.  Rehearsals  or  Dry  Runs 

A  common  mistake  In  joint  tests  is  to  assume  that  prior 
coordination  of  the  Test  Design  and  Field  Test  Plan  has  led  to  agreement  as 
to  what  Is  to  be  done.  Since  each  participating  organization,  agency,  or 
Service  has  its  own  perception  of  its  commitments,  consideration  must  be 
given  to  the  overall  accomplishment.  Consequently,  the  JTD  should  dry  run 
the  test  as  early  as  possible.  Rehearsals  down  to  the  greatest  detail 
possible  should  be  conducted  to  avoid  wasting  valuable  test  items  or 
resources. 

In  tests  involving  opposing  forces,  it  is  especially  import¬ 
ant  to  indoctrinate  JTF  members  on  the  need  to  keep  interaction  between  red 
and  blue  players  to  a  minimum  during  nontest  hours.  Failure  to  do  so  could 
lead  to  inadvertent  exchange  of  mission- related  information,  thereby  inva¬ 
lidating  trials  during  subsequent  testing. 

d.  Maintenance  of  Records 

Tests  are  conducted  to  obtain  empirical  data/information  for 
subsequent  analysis.  Test  data/information  may  be  collected  in  many  ways 
which  may  be  electronic,  electromechanical,  or  direct  observation.  When 
processed  for  analysis,  the  information  will  normally  consist  of  numbers 
(quantitative)  and  words  (qualitative).  Data  can  be  objective,  i.e.,  a 
number  obtained  by  a  measuring  device,  or  subjective,  i.e.,  obtained  from  a 
person's  estimate  or  rating.  For  example,  a  jamming  signal  could  be  objec¬ 
tively  measured  as  X  milliwatts  or  could  be  described  subjectively  as 
"severe"  or  "severity  5."  It  is  very  Important  that  information  be  corre¬ 
lated,  so  that  observations  relating  to  an  event  are  identified  with  it, 
time  tagged,  and  cued  to  each  action.  Identification  will  require  a  system 
which  labels  each  piece  of  information  accurately. 

e.  Data  Validation 

Data  validation  is  a  primary  responsibility  of  the  JTD. 
Some  joint  tests  (e.g. ,  TASVAL  and  ACEVAL/AIMVAL)  have  made  data  validation 
a  formal  process.  In  this  case,  a  data  validation  committee  formally 
reviews  the  data  against  predetermined  criteria  (objective  and/or  subjec¬ 
tive)  prior  to  its  Inclusion  in  the  data  base.  If  the  issues  in  the  test 


merit  formal  validation,  the  JTD  must  establish  the  committee  and  its  rules 
early  in  the  planning.  The  rules  should  be  coordinated  with  the  partici¬ 
pating  Services.  Data  validation  should  be  supported  by  but  be  independent 
from  the  red  and  blue  players.  There  is  often  a  temptation  to  discard  data 
which  are  obviously  faulty  or  gathered  in  the  course  of  a  test  phase  which 
was  incomplete.  The  JTD  should  adopt  as  an  operational  maxim  the  view  that 
al 1  data  should  be  retained,  regardless  of  the  perceived  quality.  Even 
"bad"  data  can  be  analyzed,  if  for  no  other  reason  than  to  determine  why 
they  were  deficient. 

f.  Calibration 

All  data  sources,  including  instrumentation,  player  equip¬ 
ment  and  timing  devices,  must  have  an  established  calibration  procedure 
which  is  followed  before  and  after  each  test  period.  Implementation  of 
this  procedure  will  minimize  data  losses  and  anomalies  which  result  from 
equipment  fluctuations.  For  example,  despite  the  theoretical  simplicity  of 
having  everyone  operate  on  the  same  time  basis,  tests  are  historically 
plagued  by  wide  time  variations  among  test  players.  An  obvious  result  of 
inattention  to  this  critical  detail  may  be  data  which  Indicates  that 
messages  were  received  before  they  were  sent  or  weapons  were  fired  before  a 
target  appeared. 

g.  Changes  During  Testing 

Before  making  changes  in  the  test  plan  for  any  reason,  test 
integrity  must  be  considered,  i.e. ,  the  original  test  design  and  the  com¬ 
parability  and  validity  of  test  results  may  be  affected.  Unless  it  can  be 
demonstrated  that  change  is  necessary  to  achieve  valid  results  and  useful 
conclusions,  it  is  inadvisable  to  make  substantive  changes  during  the 
course  of  testing.  If  an  alteration  in  the  test  plan  is  deemed  necessary 
it  must  take  the  form  of  an  addition  to,  rather  than  a  major  deviation 
from,  a  plan  which  has  been  carefully  developed. 

Historically,  some  of  the  most  valuable  findings  of  tests 
were  not  anticipated  but  became  apparent  in  the  course  of  testing.  Such 
findings  are  essentially  byproducts  of  the  planned  testing  process.  Unex¬ 
pected  test  findings  sometimes  arouse  unusual  top-level  Interest  and  test 
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modification  to  investigate  them  is  actively  supported.  Such  a  change 
should  be  made  with  an  awareness  of  the  additional  effort  it  will  cause. 
In  some  cases,  a  separate  test  --  with  all  the  urgency  the  situation 
justifies  —  may  be  considered. 

There  is  a  new  concept  currently  under  development  called 
adaptive  testing.  While  the  concept  is  not  fully  developed,  it  embodies 
useful  notions.  In  adaptive  testing,  change  is  planned  for  in  advance. 
Specifically,  the  test  matrix  is  planned  with  more  cells  than  available 
resources  can  fill.  As  the  test  proceeds,  information  gathered  during  the 
test  is  used  to  structure  the  testing  not  yet  complete.  This  may  well  be  a 
useful  concept,  but  at  the  time  of  this  writing  is  not  yet  mature, 
h.  Data  Management  During  Active  Testing 

The  primary  purpose  of  a  joint  test  is  the  collection  of 
data  to  support  the  overall  test  objective.  Consequently,  the  JTD  should 
closely  monitor  the  data  collection  and  processing  effort  to  ensure  that 
the  reduced  data  will  be  of  maximum  utility.  Successful  data  management 
begins  with  a  good  field  test  plan  and  a  comprehensive  data  management  plan 
(described  in  Sections  B~5  and  B-6  of  this  chapter).  The  growing  com¬ 
plexity  of  joint  tests  has  given  rise  to  increasingly  sophisticated  tech¬ 
niques  for  automating  the  data  collection  process.  The  JTD  should  be  aware 
of  these  techniques  and  use  them  where  possible.  If  data  collectors  are  to 
participate,  £■; . vial  1  zed  training  for  them  must  also  be  considered.  The 
JTD  should  personally  ensure  that  data  collectors  are  adequately  briefed 
prior  to  each  trial  and  monitor  all  aspects  of  the  data  collection  effort 
to  the  extent  that  he  has  a  high  degree  of  confidence  that  correct  methods 
are  being  used  and  that  valid  data  are  being  collected. 

The  JTD  should  be  aware  of  the  fact  that  data  are  being 
collected  under  dynamic  c  ditions.  It  is  not  always  possible  to  collect 
data  in  the  manner  or  in  the  location  intended.  Consequently,  the  data 
should  be  audited  continually  for  its  accuracy,  completeness,  and  time¬ 
liness.  Based  on  the  findings  of  these  audits,  procedures  should  be 
implemented  to  correct  any  deficiencies.  The  requirement  for  an  audit  is 
paramount  whenever  significant  deviations  occur  (such  as  changes  in  test 
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location  or  personnel).  The  JTD  should  be  aware  that,  once  the  test  execu¬ 
tion  phase  is  underway,  changes  in  the  test  should  be  carefully  evaluated 
due  to  the  serious  ramifications  they  may  have  on  the  data  management 
effort. 

If  the  JTD  must  await  validated  and  processed  data,  he  will 
be  unable  to  exercise  timely  control.  Thus,  all  data  management  schemes, 
automated  or  manual,  should  have  a  quick  look  feature  to  permit  preliminary 
assessments  concerning  test  trial  validity,  test  status,  and  overall 
trends.  Used  as  a  feedback  to  test  control,  quick  look  data,  albeit  not  as 
high  in  confidence  as  final  processed  data,  can  alert  the  JTD  to  problems 
in  players,  tactics,  rules  of  engagement,  or  other  areas  and  enable  him  to 
act  appropriately.  Quick  look  can  also  minimize  the  time  required  for 
preparation  of  the  final  report. 

i.  Player  Force  Maturation  and  Learning  Effects 

In  some  tests,  it  is  possible  for  significant  positive 
learning,  training,  or  maturation  to  occur.  The  player  can  be  exposed  to 
unfamiliar  training  conditions  (for  example,  intensive  jamming).  Per¬ 
forming  the  test  replications  may  have  enough  training  value  to  signif¬ 
icantly  improve  player  performance.  Such  data  should  not  be  averaged,  as 
the  value  of  realistic  training  may  be  inferrable  from  the  improvement  in 
performance.  It  is  equally  possible  that  the  dedication  and  competitive 
spirit  of  players  can  result  in  negative  learning  (or  a  propensity  to  "beat 
the  test").  The  JTD  and  his  staff  must  be  alert  to  these  changes  and  keep 
the  test  structured  so  that  it  is  realistic  and  impartial. 

0.  ANALYSIS 

1 .  Requirements 

Requirements  for  JT&E  program  analysis  exist  throughout  the 
entire  test  process— from  inception  and  initial  planning  to  reporting  of 
final  results.  Although  JT&E  programs  are  widely  diversified  and  vary 
greatly  in  overall  scope  and  objectives  from  program  to  program,  analysis 
is  involved  in  the  main  phases  of  any  individual  test  program.  In  a  broad 
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sense,  the  main  phases  of  a  test  program  include  the  Pre-Test  Phase,  Active 
Test  or  Execution  Phase,  and  the  Post-Test  Phase.  Each  of  these  phases  has 
its  own  relevant  types  of  analysis  requirements. 

a.  Pre-Test  Requirements 

As  discussed  in  Section  B  of  this  chapter,  the  analytical 
techniques  used  to  evaluate  test  results  are  closely  tied  to  the  statisti¬ 
cal  test  design  and  the  Data  Management  Plan.  Thus,  the  pre-test  analysis 
requirement  is  essentially  to  ensure  consistency  among  the  techniques  to  be 
used,  the  design  of  the  test,  and  the  data  to  be  collected.  A  useful 
approach  is  to  actually  "dry  run"  all  anticipated  analyses  using  dummy  data 
of  the  type  expected  during  active  testing.  Gaps  or  weak  assumptions 
should  then  become  apparent  before  testing  begins.  This  pretesting  should 
apply  to  al 1  data  expected,  qualitative  as  well  as  quantitative. 

b.  Active  Test  Requirements 

During  the  active  testing  phase,  the  intended  objective  is 
to  conduct,  follow  and  complete  test  execution  according  to  the  test  plan. 
However,  unforeseen  problems  and  difficulties  may  occur  and  decisions  must 
be  made.  Each  trial  of  a  JT&E  test  will  generally  result  in  a  large  amount 
of  data  relating  to  vehicle  positions,  weapon  events,  fire  control  systems, 
and  environmental  conditions.  It  is  Important  to  the  successful  conduct  of 
the  test  that  a  limited  evaluation  of  an  appropriate  cross  section  of  these 
data  be  performed  as  soon  as  possible  for  each  trial  and/or  subsets  of  the 
trial.  This  evaluation  is  the  quick-look  analysis  and  is  based  on  specific 
techniques  prepared  in  advance. 

Quick-look  analysis  provides  for  substantiation  of  a  suc¬ 
cessful  trial.  In  the  case  of  failure,  it  identifies  marginal  operation 
and  failure  of  specific  items  of  equipment,  and  avoids  costly  processing  of 
invalid  data.  When  a  trial  results  in  failure,  the  quick-look  analysis 
aids  in  minimizing  the  time  to  correct  failure  causes.  The  Invalid  trial 
can  then  be  repeated.  Additionally,  quick-look  analysis  may  show  that 
actual  system  performance  deviates  considerably  from  that  anticipated.  If 
this  should  occur,  the  test  design  matrix  can  be  modified,  as  appropriate, 
early  during  the  test  sequence  to  provide  more  useful  data  for  the  final 
analysis  of  the  system  undergoing  test  and  evaluation. 
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c.  Post-Test  Requirements 

The  basic  objective  of  analysis  after  program  operations  are 
completed  is  to  quantify  measures  of  performance  and  effectiveness  for  the 
scenarios/threats  investigated  and  to  indicate  the  degree  of  confidence  in 
the  reported  quantities.  Measurements  taken  during  operations  and  the 
resultant  data  processing  are  the  basis  for  subsequent  analysis.  Experi¬ 
ence  has  shown  that  unexpected  problems  are  often  encountered  in  this  area. 
Some  types  of  data  may  or  may  not  be  recoverable.  Decisions  must  be  made 
at  various  stages.  Decisions  can  range  from  wholesale  dropping  of  some 
types  of  data  because  of  lack  of  validity  to  total  acceptance  of  all  data 
for  certain  runs  or  trials.  Since  data  processing  is  the  beginning  of 
confidence  or  lack  of  confidence  in  the  final  results,  it  is  an  inherent 
part  of  the  analysis. 

Processed  data  are  analyzed  according  to  statistical  tech¬ 
niques  generally  contained  in  the  test  design  and/or  analysis  plans  with 
any  appropriate  excursions.  The  initial  procedure  in  statistical  analysis 
is  to  measure  and  adjust  or  standardize  for  extraneous  effects  such  as 
changes  in  environment,  the  effect  of  learning  or  practice,  and  other 
factors.  Formal  statistical  techniques  are  used  to  determine  if  a  rela¬ 
tionship  among  variables  exists,  what  the  relationship  is,  and  the 
importance  of  the  relationship.  This  is  the  basis  for  standardizing  or 
normalizing  the  data  for  additional  analysis. 

In  certain  cases,  it  may  be  possible  to  combine  data  from 
different  test  scenarios.  The  approach  is  to  begin  with  the  smallest  ele¬ 
ment,  a  single  test  scenario  or  test  condition,  and  combine  it  with  as  many 
others  as  are  valid.  The  basis  for  limiting  the  combination  of  the 
different  sets  is  again  formal  statistical  techniques.  When  these  techni¬ 
ques  indicate  significant  differences,  a  pooling  or  combining  of  the  data 
is  unwarranted,  and  separate  values  of  performance  and  effectiveness  mea¬ 
sures  are  required  for  the  different  scenarios. 

2.  Techniques 

Although  no  two  tests  are  ever  identical,  data  analysis  problems 
can  often  be  reduced  to  a  relatively  short  list.  Each  test  will  have  its 
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own  problems,  and  in  certain  instances,  the  unique  nature  of  the  test 
design  or  the  test  data  will  require  the  statistician  to  employ  specialized 
and  seldom-used  analysis  techniques.  However,  there  are  usually  a  limited 
number  of  fundamentally  different  questions  being  asked:  What  is  the  value 
of  some  parameter  believed  to  be?  In  what  range  will  the  value  of  some 
parameter  probably  fall?  What  is  the  relationship  between  values  of  one 
variable  and  values  of  another?  What  are  the  sources  of  the  observed  vari¬ 
ability  among  trials?  Does  a  parameter  have  the  value  claimed  by  the  advo¬ 
cate  or  developer  of  the  system? 

At  times  these  same  questions  can  be  turned  around  to  ask  about 
the  probability  of  realizing  certain  conditions  in  the  population  of  infer¬ 
ence.  What  is  the  probability  that  a  parameter  is  between  two  given 
values?  What  is  the  probability  that  the  parameter  is  less  than  a  given 
value?  What  is  the  probability  that  the  same  percent  of  the  population 
values  for  a  variable  will  be  less  than  a  given  value? 

A  brief  discussion  of  four  major  analytical  techniques  is  pre¬ 
sented  in  this  section.  While  this  discussion  does  not  exhaust  the  list  of 
potential  analytical  approaches,  most  of  the  problems  the  JTD  can  expect  to 
encounter  can  be  handled  by  the  techniques  discussed  in  this  manual. 

a.  Statistical  Estimation  Techniques 

There  are  several  types  of  statistical  estimation  techniques 
which  can  be  employed  in  the  analysis  of  JT&E  data.  These  can  include 
point  estimates,  interval  estimates,  and  interval  estimates  of  content  or 
tolerance  intervals.  This  type  of  analysis  involves  the  estimation  of 
properties  of  a  total  population  on  the  basis  of  the  properties  discovered 
about  a  random  sample  from  that  population. 

b.  Comparison  Techniques 

Often  it  is  useful  to  compare  the  properties  of  one  set  of 
observations  with  those  of  an  "ideal"  or  assumed  set,  or  one  or  more  sets 
of  observations  made  under  different  conditions.  Comparison  techniques 
such  as  tests  of  significance,  analysis  of  variance,  and  tests  of  indepen¬ 
dence  are  useful  in  such  instances. 
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c.  Graphic  Techniques 

In  many  instances,  a  great  deal  can  be  learned  from  test 
data  when  it  is  drawn  on  paper  in  “picture"  or  graphic  form.  As  the  amount 
of  data  collected  grows,  the  process  of  assimilating  the  data  becomes  more 
difficult.  A  large  table  of  numbers  may  be  inconclusive  and  the  statisti¬ 
cal  descriptors  chosen  to  summarize  the  data  may  not  tell  the  whole  story. 
A  visual  presentation  of  the  data,  not  totally  dependent  on  numbers,  may 
suggest  trends  and  exceptions  not  immediately  obvious  from  the  raw  data  or 
statistical  tests.  It  can  be  helpful  both  to  the  data  analyst  and  to  the 
user  of  a  test  report. 

d.  Nonparametric  Techniques 

Situations  can  arise  in  JT&E  data  analysis  which  preclude 
the  use  of  parametric  statistics.  It  could  be  the  case  that  there  is  no 
knowledge  of  the  distribution  of  the  variable  under  examination;  alter¬ 
nately,  it  could  be  the  case  that  the  distribution  is  definitely  not  of  the 
required  type  for  applying  the  desired  theory.  It  is  also  possible  that 
some  variables  of  interest  will  not  be  measured  on  an  interval  scale.  For 
example,  a  target  hit  (or  miss)  is  a  dichotomous  variable  since  it  can  only 
be  characterized  by  a  discrete  value.  Similarly,  subjective  rankings  of 
various  ECM  or  ECCM  techniques  comprise  an  ordinal  scale  and  are  not  suit¬ 
able  for  analysis  using  parametric  techniques. 

Certain  techniques  have  been  designed  to  replace  standard 
tests  when  the  assumptions  and  requirements  of  those  tests  such  as  normal¬ 
ity  and  interval  scales  of  measurements  are  not  satisfied.  These  are  non¬ 
parametric  techniques  since  they  do  not  test  parameter  values  of  known 
population  types.  Probably  the  most  widely  used  nonparametric  technique  is 
the  Chi-square  test  for  independence  and  goodness  of  fit.  Additional 
nonparametric  techniques  which  can  be  employed  in  analysis  of  JT&E  data 
include: 

e  Wilcoxon  Matched- Pairs,  Signed- Ranks  Test 
e  Coefficient  of  Concordance  (Kendall's  W) 
e  Friedman  Two-Way  Analysis  of  Variance 
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•  Kolmogorov- Smirnov  Two- Sample  Test 

•  Wilcoxon  Rank-Sum  Test. 

The  preceding  list  is  by  no  means  exhaustive  since  other 
nonparametric  techniques  exist.  However,  those  listed  are  among  the  most 
likely  for  use  in  JT&E  data  analysis. 

E.  POST-TEST  ACTIVITIES 

1 .  Test  Reporting 

a.  Introduction 

The  primary  purpose  of  test  reporting  is  to  communicate  the 
results  of  the  test.  While  this  might  seem  obvious  at  first,  it  is  essen¬ 
tial  that  all  involved  in  preparing  reports  understand  that  the  stress  is 
placed  on  the  single  word--communicate.  Not  only  must  test  reports  be 
timely,  factual,  concise,  complete,  and  accurate,  they  must  also  be  written 
for  their  intended  audience.  Generally,  this  audience  is  nontechnical. 
However,  it  is  essential  that,  while  written  primarily  in  nontechnical 
terms,  test  reports  contain  sufficient  data  and  analysis  to  support  the 
recommendations  and  conclusions. 

At  a  minimum,  the  JTD  is  responsible  for  submitting  the  fol¬ 
lowing  reports  to  the  ODTE  for  evaluation  and  approval:  interim  reports, 
special  reports  (as  required),  a  JTD  Management  Report,  and  a  JTF  Final 
Test  Report.  These  reports  are  discussed  briefly  in  this  section. 

b.  Interim  Reports 

Interim  Reports  are  used  to  communicate  important  test 
findings,  milestones,  progress,  or  problems  prior  to  the  completion  of  the 
test.  Consequently,  no  hard  and  fast  rules  exist  for  the  number  of  Interim 
Reports  to  be  submitted,  nor  for  their  content,  format,  or  means  of  trans¬ 
mission.  These  will  vary  according  to  the  complexity  of  the  test,  the 
interest  shown  in  it,  and  the  importance  attached  to  the  information  it  is 
expected  to  provide. 
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The  JTD  should  exercise  caution  when  preparing  an  Interim 
Report.  Since  the  test  will  not  be  complete,  there  will  probably  be  insuf¬ 
ficient  data  on  which  to  base  valid  conclusions.  Yet,  users— and  even 
members  of  the  JTF — might  be  tempted  to  "jump  to  conclusions"  on  the  basis 
of  an  Interim  Report,  especially  if  the  test  is  at  all  controversial. 
Caveats  on  the  data  presented  are  essential. 

c.  Special  Reports 

Occasionally,  the  JTD  may  prepare  a  Special  Report  concern¬ 
ing  a  major  program  milestone,  significant  data  collected  to  date,  or  a 
critical  testing  problem  which  requires  immediate  attention.  In  addition, 
users  may  request  data  which  have  been  generated  up  to  a  given  point  in  the 
test.  Similarly,  the  DOTE  may  require  a  special  status  report  on  the  test. 
In  any  case,  the  JTD  should  exercise  the  same  caution  in  preparing  Special 
Reports  that  he  would  for  Interim  Reports  and  caveat  all  points  where 
insufficient  data  and  incomplete  testing  could  lead  to  erroneous 
conclusions. 

d.  JTD  Management  Report 

The  JTO  Management  Report  is  an  "after  action"  report  of  the 
test  and  its  objectives.  It  is  the  vehicle  for  transmitting  lessons 
learned,  the  potential  utility  of  test  results,  and  conclusions  and  recom¬ 
mendations.  This  report  is  used  by  the  JTD  to  report  on  management, 
administrative,  and  testing  issues  which  will  be  useful  to  other  JTDs.  It 
is  broader  in  scope  than  the  traditional  "lessons  learned"  report  since  the 
JTD  is  also  required  tc  assess  the  overall  utility  of  the  test  objectives 
and  results  and  to  report  his  own  conclusions  and  recommendations. 

The  Management  Report  is  submitted  to  the  DOTE  for  inclusion 
in  the  JT&E  information  file  and  distribution  to  interested  users.  There 
is  no  fixed  format;  however,  the  following  items  should  be  covered: 
e  Overview  of  Test  Report 

e  Assessment  of  test  objectives 

•  Lessons  learned 

•  Potential  utility  of  test 

•  Recommendations  and  conclusions. 
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JTF  Final  Test  Report 


The  Final  Test  Report  is  p* spared  by  the  Joint  Test  Force 
and  provides  the  permanent  record  of  the  results  of  the  test.  This  report 
documents  the  factual  aspects  of  the  test  such  as  environmental  test  condi¬ 
tions,  scenarios  played,  data  collected,  preliminary  empirical  results, 
test  participants,  and  any  related  items  or  comments  relevant  to  the 
assessment  of  test  results.  While  the  DDTE  may  contract  for  an  independent 
evaluation,  the  JTD  retains  responsibility  for  conducting  necessary  analy¬ 
ses  and  providing  the  JTF's  evaluation  of  test  results. 

The  Final  Test  Report  is  submitted  as  a  bound,  final  docu¬ 
ment  to  the  DOTE  for  distribution  to  interested  users.  It  will  contain  the 
following  major  sections: 

e  Purpose  of  the  Test 

e  Method  of  Accomplishment 

e  Discussion  and  Analysis 

•  Conclusions  and  Recommendations 

•  Annexes: 

-  Detailed  Test  Description 
Test  Environment 
Test  Methodology  and  Design 

Test  Data,  Statistical  Summaries,  and  Findings  of  Fact 
Supporting  Data  and  Analysis  (including  results  of  quick 
look  analysis) 

Test  Organization 
Test  Operation 
Maintenance 
Logistics  and  Supply 
Personnel 


Training 

Safety 

Security 

Funding 

Materiel  Disposition. 
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Materiel  Accounting  and  Disposition 

The  conduct  of  joint  testing  sometimes  requires  procurement  of 
unique  instrumentation  using  funds  from  PE  658040.  Such  investment  in 
unique  instrumentation  or  equipment  has  been  and  most  likely  will  continue 
to  be  substantial.  Therefore,  a  maximum  return  on  this  investment  must  be 
sought  through  proper  utilization  and  effective  management. 

a.  Pol  icy 

Unique  instrumentation  procured  for  joint  testing  may  be  of 
a  highly  advanced  technical  design,  or  may  permit  significantly  improved 
evaluation  of  military  systems  and  tactics.  For  these  reaons,  such  instru¬ 
mentation  should  be  of  considerable  benefit  to  the  military  Services  in  the 
conduct  of  their  own  test  and  evaluation  programs.  It  will  be  the  policy 
of  the  DDTE  to  turn  over  custody  of  joint  test  instrumentation  to  the 
military  Services  once  such  instrumentation  is  no  longer  being  used  full 
time  for  joint  test  purposes.  This  policy  applies  to  unique  JT&E  equipment 
as  well. 

It  is  necessary  that  an  orderly  procedure  be  followed  in 
turning  over  joint  test  instrumentation  and  equipment  to  the  Services. 
This  orderly  procedure  will  normally  involve  the  following  steps: 

•  Conduct  of  a  Utilization  Study  to  determine  the  best  use  for  the 
instrumentation  or  equipment 

•  Development  and  signature  of  a  Memorandum  of  Understanding 
between  the  DOTE  and  the  custodian  Service 

•  Development  by  the  custodian  Service  and  approval  by  the  DDTE  of 
a  utilization  plan  for  the  instrumentation  or  equipment. 

Once  custody  of  joint  test  instrumentation  and  equipment  has  been  turned 
over  to  a  Service,  it  will  remain  the  responsibility  of  the  DDTE  to  verify 
that  such  instrumentation  or  equipment  is  properly  and  fully  utilized. 

b.  Procedures 

This  procedure  will  apply  to  all  major  items  of  instrumenta¬ 
tion  and  equipment  procured  or  ultimately  financed  either  fully  or  partly 
with  funds  from  Program  Element  65804D,  Test  and  Evaluation.  A  major  item 


III -62 


of  instrumentation  is  defined  as  an  equipment,  or  an  aggregation  of  equip¬ 
ments  to  perform  a  single  function,  whose  total  procurement  cost  exceeds 
$50,000.  This  procedure  will  be  waived  only  with  the  concurrence  of  the 
Director  Defense  Test  and  Evaluation. 

(1 )  Utilization  Study 

A  utilization  study  for  a  particular  item  of  instrumen¬ 
tation  or  equipment  will  normally  be  conducted  approximately  nine  months 
prior  to  the  estimated  date  of  release  to  the  Service.  The  purpose  of  the 
utilization  study  is  to  provide  a  basis  for  determining  the  best  possible 
usage  for  the  equipment.  If  necessary,  the  study  could  be  conducted  by  an 
independent  agency  (e.g. ,  IDA)  with  inputs  from  other  organizations  as 
required.  The  utilization  study  should  accomplish  the  following: 

•  Develop  a  list  of  all  potential  applications  for  the  instrumenta¬ 
tion  or  equipment 

•  Consider  the  capabilities  and  limitations  of  other  instrumenta¬ 
tion  or  equipment  for  the  same  applications 

•  Determine  the  cost  effectiveness  of  the  instrumentation  or  equip¬ 
ment  for  particular  applications 

•  Specify  modifications  or  changes  required  for  particular  applica¬ 
tions 

•  List  possible  applications  in  order  of  priority  according  to 
urgency  of  requirements  and  cost  effectiveness  for  particular 
applications. 

(2)  Memorandum  of  Understanding 

The  principal  mechanism  for  turning  over  custody  of 
joint  test  instrumentation  or  equipment  to  a  military  Service  will  be  a 
Memorandum  of  Understanding  signed  by  the  DOTE  and  an  appropriate  represen¬ 
tative  of  the  Service.  The  Memorandum  of  Understanding  (MOU)  will  contain 
the  following  essential  elements: 

•  Designation  of  the  parties  to  the  MOU 

•  Delegation  authority 

•  Identification  of  a  specific  Service  Agency  to  act  as  custodian 


for  the  equipment 
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•  Detailed  identification  of  equipment  to  be  covered  by  the  MOU 
e  Statement  of  priority  for  JT&E  usage 

e  Statement  of  availability  to  all  Services 
e  Statement  of  funding  procedures 

•  Instructions  on  equipment  modifications  to  be  permitted 

e  Provision  for  submission  of  a  utilization  plan  on  an  annual  or 
more  frequent  basis  if  required 
e  Statement  of  conditions  for  deactivation  of  equipment 

•  Designation  of  effective  date  for  MOU. 

(3)  Utilization  Plan 

The  Service  Agency  having  custody  of  the  Joint  Test 
equipment  will  prepare  and  submit  annually  a  utilization  plan  for  the 
equipment.  The  utilization  plan  shall  contain  the  following: 

•  A  listing  of  all  requests  received  for  equipment  utilization 

•  A  listing  of  currently  programmed  utilization  tasks  and  dates  for 
their  accomplishment 

e  A  description  and  schedule  for  programmed  maintenance,  modifica¬ 
tion,  and  update  actions 
e  A  description  of  equipment  funding  status, 
c.  Implementation 

The  Director  Defense  Test  and  Evaluation  will  designate 
which  Service  is  to  assume  custody  for  specific  items  of  instrumentation  or 
equipment  and  sign  MOUs  for  specific  items  of  instrumentation  or  equipment. 
He  will  also  approve  instrumentation  or  equipment  utilization  plans. 

The  Oeputy  Oi rector  Tactical  Air  and  Land  Warfare  Systems 
(DOTE)  will  sponsor  and  administer  MOUs  and  consolidate  and  submit  annually 
for  DOTE  approval  the  utilization  plan  for  JT&E  instrumentation  or  equip¬ 
ment.  The  Deputy  Director  for  Test  Facilities  and  Resources  (DOTE)  will 
monitor  funding  arrangements  and  implement  DOTE  funding  guidance  for  JT&E 
instrumentation  or  equipment  covered  by  MOUs.  In  addition,  he  will  provide 
technical  assistance  In  developing  and  evaluating  Instrumentation  alter¬ 
natives  and  utilization  of  existing  test  resources. 
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Joint  Test  Directors  will  conduct  utilization  studies  for 
instrumentation  and  equipment  procured  under  their  respective  Joint  Tests, 
as  necessary.  They  will  also  prepare  and  coordinate  MOUs  for  instrumenta¬ 
tion  and  equipment  procured  under  their  respective  Joint  Tests,  and  in  the 
case  of  instrumentation  equipment  assigned  to  a  joint  test,  provide  inputs 
for  the  annual  update  of  the  overall  utilization' plan. 

3.  Personnel  Actions 

The  close-out  of  a  JTF  should  be  treated  in  the  same  manner  as 
the  deactivation  of  any  military  unit.  If  conducted  properly,  with  care 
for  the  morale  and  general  welfare  of  personnel,  the  closing  out  JTF  will 
continue  to  function  efficiently.  The  first  rule  is  to  plan  early  ar.d 
thoroughly.  As  in  unit  deactivations,  good  plans  and  follow-through  will 
anticipate  the  important  close-down  activities  and  schedule  personnel 
departures  so  that  major  events  are  not  disrupted.  In  order  to  assure  that 
awards  planning  is  conducted  properly,  an  awards  and  decorations  officer 
should  be  appointed.  Specific  timing  Will  vary  from  test  to  test,  but  in 
general  the  bulk  of  the  award  packages  should  be  prepared  before  the  final 
phase-down  begins. 

Planning  must  include  interaction  with  Service  personnel  acti¬ 
vities  to  assure  release  dates  that  are  consistent  with  professional 
close-out  of  the  JTF  and  not  prejudicial  to  the  personnel  themselves.  As 
the  pace  of  close-out  picks  up,  coping  with  all  of  the  necessary  milestones 
without  a  thorough  plan  can  lead  to  chaos.  More  important,  for  the  person¬ 
nel  who  have  made  the  JT&E  a  success,  a  smooth  close-out  should  be  the  cul¬ 
mination  of  a  unique  and  stimulating  assignment. 

4.  Data  Repository 

At  the  conclusion  of  any  test,  a  major  problem  is  the  disposition 
of  the  large  quantity  of  raw  data  and  the  official  data  file.  If  a  compre¬ 
hensive  data  management  plan  was  implemented,  data  disposition  and  handling 
will  have  been  addressed.  The  data  will  be  well  organized  and  labelled  and 
documentation  will  exist  which  specifies  how  the  various  data  elements  can 
be  retrieved.  Thus,  the  disposition  problem  can  be  reduced  to  one  of 
arranging  for  proper  long-term  storage  from  which  the  data  can  be  accessed 
with  minimal  difficulty. 
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Data  will  have  been  collected  by  many  different  means  and  will  be 
in  many  different  formats.  Thus,  there  will  be  forms,  computer  tapes, 
punched  cards,  reports,  and  other  physical  forms  which  must  be  stored.  The 
complexity  of  the  repository  problem  will  vary  with  the  size  of  the  test. 
Further,  there  will  be  a  problem  associated  with  assigning  the  burden  of 
storing  the  data  to  one  or  more  DoD  components. 

Generally,  the  Lead  Service  will  be  responsible  for  maintaining 
the  OSD-funded  data  repository.  The  JTD  should  identify  as  early  in  the 
test  cycle  as  possible  the  organization  within  the  Lead  Service  which  will 
provide  the  storage  and  retrieval  facilities.  These  facilities  will  range 
from  several  file  drawers  to  environmentally  controlled  computer  tape 
storage,  to  on-line  storage  in  a  computer  center.  The  JTD  should,  on  the 
basis  of  his  data  management  plan,  anticipate  the  kinds  of  storage  facili¬ 
ties  required  and  initiate  preliminary  arrangements  early  in  the  test. 

If  more  than  one  storage  point  will  be  required,  the  location  and 
method  of  access  to  each  should  be  documented  and  made  part  of  the  JTF 
Final  Test  Report  and  theJTD  Management  Report.  In  some  cases,  data  will 
be  stored  in  more  than  one  location  by  more  than  one  Service  or  Agency. 
The  JTD  should  anticipate  the  long  lead  time  needed  to  resolve  the 
coordination  problems  inherent  in  such  a  storage  arrangement  so  that 
procedures  for  accessing  the  data  can  be  documented  by  the  time  the  test  is 
over  and  the  repositories  are  created.  In  cases  where  a  Service  or  Agency 
other  than  the  Lead  Service  will  have  some  or  all  data  repository 
responsibilities,  the  DOTE  should  be  involved  in  all  discussions  and 
agreements  concerning  the  establishment  of  the  repository. 

Whatever  the  physical  form(s)  and  location(s)  of  the  repository, 
the  JTD  must  document  the  following  information  for  all  users: 

•  The  form  of  the  data 

e  How  the  data  are  classified 

•  Where  the  data  are  stored 

•  How  the  data  can  be  accessed 

•  Restrictions/problems  in  the  use  of  the  data 

•  How  long  the  data  will  be  stored. 
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CHAPTER  IV 
SPECIAL  TOPICS 

A.  SERVICE  SUPPORT  MECHANISMS 

1 .  Introduction 

The  process  of  nominating,  selecting,  chartering,  designing, 
planning,  and  executing  a  Joint  Test  &  Evaluation  involves  many  partici¬ 
pants.  In  each  test,  the  JTD  must  establish  good  communications  with  the 
DDTE,  the  Lead  Service,  and  each  supporting  Service.  In  this  section,  the 
Service  organizations  are  described  so  the  JTD  may  see  the  framework  of  the 
formal  organization  through  which  he  and  his  staff  must  operate. 

For  the  new  JTD,  the  formal  points  of  contact  will  provide  an 
initial  entry  into  Service  Headquarters.  However,  in  every  case  of  a 
successful  test  in  the  past,  the  JTD  and  his  staff  established  informal 
relationships  which  served  to  supplement  and  enrich  the  formal  contacts. 
Successful  establishment  of  working  relationships  will  provide  the  JTD  with 
the  means  of  obtaining  necessary  Service  support. 

2.  United  States  Army 

a.  Test  Schedule  and  Review  Process 

The  Test  Schedule  and  Review  Committee  (TSARC),  established 
by  the  Army  Chief  of  Staff  (CSA),  provides  high-level  centralized  resource 
management  for  user  testing  in  the  Army.  This  centralized  management  maxi¬ 
mizes  the  use  of  limited  resources  and  minimizes  the  adverse  impact  on 
operational  readiness.  Composition  of  the  TSARC  is  shown  in  Table  IV-1. 

The  TSARC  meets  in  June  and  December,  although  the  Chairman 
may  call  special  meetings  to  address  problems  which  require  more  immediate 
attention.  Test  resource  allocations  are  developed  within  existing  budget 
and  program  constraints  in  accordance  with  Army  priorities  for  the  current 
and  budget  years.  These  allocations  and  planning  guidance  for  the  out- 
years  are  published  in  a  tasking  document  known  as  the  Five-Year  Test 
Program  (FYTP).  All  new  entries  and  proposed  changes  in  the  FYTP  are 
submitted  by  TSARC  members  to  the  U.S.  Army  Operational  Test  and  Evaluation 
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TABLE  IV- 1.  TSARC  COMPOSITION 

CHAIRMAN:  CG  OPERATIONAL  TEST  AND  EVALUATION  AGENCY  (OTEA). 

GENERAL  OFFICER  REPRESENTATIVES: 

•  DCS/RESEARCH,  DEVELOPMENT  AND  ACQUISITION 

•  DCS/OPERATIONS  ANO  PLANS 

•  DCS/LOGISTICS 

•  OCS/PERSONNEL 

•  COMPTROLLER  OF  THE  ARMY 

•  ASSISTANT  CHIEF  OF  STAFF  AUTOMATION/COMMUNICATIONS  (ACSAC) 

•  THE  SURGEON  GENERAL 

•  CHIEF  STUDY  PLANS  MANAGEMENT  OFFICE  (SPMO) 

•  TRAINING  AND  DOCTRINE  COMMAND  (TRADOC) 

•  MATERIEL  DEVELOPMENT  AND  READINESS  COMMAND  (DARCOM) 

•  FORCES  COMMAND  (FORSCOM) 

•  COMMUNICATIONS  COMMANO  (USACC) 

•  INTELLIGENCE  ANO  SECURITY  COMMAND  (tNSCOM) 

OTHER  REPRESENTATIVES: 

TRADOC  MAY  PROVIDE  REPRESENTATION  FROM 

•  COMBAT  DEVELOPMENTS  EXPERIMENTATION  COMMAND  (CDEC) 

•  COMBINED  ARMS  CENTER  (CAC) 

•  TRADOC  COMBINED  ARMS  TEST  ACTIVITY  (TCATA) 

•  LOGISTICS  CENTER  (USALOGC) 

DARCOM  MAY  PROVIDE  REPRESENTATION  FROM 

•  TEST  AND  EVALUATION  COMMAND  (TECOM) 

OTHER  ARMY  STAFF  AGENCIES  AND  COMMANDS  (BY  INVITATION  OF  THE 
CHAIRMAN). 

NOTE:  THE  TSARC  WILL  BE  SUPPORTED  BY  AN  0-6/0- 5  (OR  EQUIVALENT  DA 

CIVILIAN)  WORKING  GROUP  WITH  REPRESENTATION  FROM  THE 
COMMANDS  ANO  AGENCIES  REPRESENTED  ON  THE  TSARC. 

SOURCE:  U.S.  ARMY  REGULATION  71-3 

8  MARCH  1977 
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Agency  (OTEA)  for  TSARC  approval.  Deviations  in  resource  allocations 
resulting  from  program  or  budget  decisions  are  reported  to  OTEA  and  dissem¬ 
inated  to  the  commands  represented  on  the  TSARC.  Approval  of  the  FYTP  by 
the  Deputy  Chief  of  Staff  for  Operations  and  Plans  (DCSOPS)  is  required 
before  its  publication. 

Definition  of  test  resource  requirements  is  provided  by  the 
Outline  Test  Plan  (OTP).  For  a  Joint  Test  and  Evaluation  with  Army  partic¬ 
ipation,  the  OTP  is  prepared  in  draft  by  the  JTD  and  his  staff  and  final¬ 
ized  by  OTEA. 

To  summarize,  the  responsibilities  of  the  TSARC  apply  to  all 
phases  of  Army  participation  in  JT&E.  Those  responsibilities  are  to: 
e  Review  and  recommend  coordinated  OTP  for  inclusion  in  the  FYTP 
e  Review  and  recommend  test  priorities  for  user  testing 
e  Review  and  coordinate  resources  for  support  of  user  testing 

•  Resolve  conflicts  between  test  requirements  and  other  missions 
e  Review  funding  for  user  tests 

e  Review  and  recommend  approval  of  the  FYTP  to  the  DCSOPS. 
b.  Specific  Responsibilities 

(1)  Headquarters  Level 

The  point  of  contact  within  the  Headquarters,  United 
States  Army,  is  the  DCSOPS.  An  abbreviated  Department  of  the  Army  (DA) 
Staff  organization  chart  is  shown  in  Figure  IV- 1.  Within  the  DCSOPS  there 
is  a  Test  Coordination  Office  (DAMO-RQT)  which  has  both  a  HQDA  Staff  offi¬ 
cer  and  an  Operational  Test  and  Evaluation  Agency  (OTEA)  liaison  officer. 
Considerable  interaction  takes  place  between  the  HQDA  Staff  and  OTEA 
regarding  JT&E  matters. 

The  Requirements  Directorate  (DAMO-RQ)  of  the  ODCSOPS 
is  the  point  of  contact  for  JT&E  matters  in  the  HQDA  Staff.  As  such 
DAMO-RQT: 

t  Staffs  Army  JT&E  nominations 

e  Coordinates  on  feasibility  studies,  test  designs,  and  test  plans 

•  Provides  USA  liaison  to  OSD/DDTE  on  issues  of  JT&E  policy  and 
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Figure  IV-1 .  Department  of  the  Army  Staff  Organization 
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•  Provides  a  USA  member  to  the  JT&E  Planning  Committee  and  Senior 
Advisory  Council 

•  Staffs  approval  of  the  FYTP 

•  Staffs  nomination  of  Army  JTD  and  selection  of  Army  DTD 

•  Staffs  resource  issues  for  JT&E 

•  Coordinates  personnel  allocations  for  JT&E 

•  Coordinates  on  letters  of  instruction  to  Army  JTD/DTD. 

(2)  Field  Level 

OTEA  is  the  Army's  independent  test  agency  which 
reports  to  the  Chief  of  Staff.  It  is  located  in  Falls  Church,  Virginia, 
and  is  the  Army's  center  for  user  testing  (operational  T&E,  JT&E,  force 
development  test  and  experimentation).  Within  existing  policy  and  resource 
constraints,  OTEA  has  overall  Army  management  responsibility  for  JT&E  and 
is  the  Army  point  of  contact  with  OSD  for  specific  JT&Es-.  As  such  OTEA: 

e  Staffs  and  prepares  the  Army  position  on  test  documents  (feasi¬ 
bility  studies,  designs,  plans) 

•  Staffs  the  tasking  of  Army  commands  and  agencies  conducting  or 
supporting  JT&E 

•  Recommends  the  Army  proponent,  i.e. ,  the  organization  responsible 
for  providing  doctrina1  and  technical  expertise  required  to 
support  the  test 

e  Provides  a  USA  member  to  the  JT&E  Planning  Committee  and  Senior 
Advisory  Council 

•  Prepares  letters  of  instruction  for  the  Army  JTD/DTD.  (Outlines 
specific  Army  responsibilities,  reporting  channels  within  the 
Army,  points  of  contact,  and  necessary  administrative 
instructions) 

•  Recommends  for  DCSOPS  approval ,  Army  JTD/DTD  (drawn  from  Army¬ 
wide  resources). 

3.  United  States  Navy 

a.  Chief  of  Naval  Operations  (CNO)  Staff  Level 

The  Navy  Point  of  Contact  for  JT&E  is  located  in  the  Office 
of  Research,  Development,  Test,  and  Evaluation  (0P098).  One  major 
subdivision  of  0P098  is  0P983,  the  Test  and  Evaluation  Division.  The 
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Commander  of  the  Navy’s  independent  operational  test  agency,  the  Opera¬ 
tional  Test  and  Evaluation  Force  (0PTEVF0R),  in  Norfolk,  Virginia,  is  also 
the  Director  of  the  Test  and  Evaluation  Division  (0P983).  The  JT&E  point 
of  contact  is  0P983E  (see  Figure  IV-2)  who  provides  liaison  with  OSD  in 
JT&E  matters.  However,  for  a  specific  test  with  Navy  participation,  the 
CNO  staff  selects  an  appropriate  sponsor.  Generally,  the  sponsor  will  be 
from  that  part  of  the  CNO  staff  which  has  the  major  functional  interest  in 
the  test.  For  example,  a  test  involving  Naval  Aviation  resources  would 
probably  be  sponsored  in  0P05,  the  DCNO  (Air  Warfare).  Therefore,  in  the 
CNO  staff,  the  actions,  responsibilities,  and  staffing  locations  are  shared 
between  0P983E  and  a  sponsor  for  each  test  with  Navy  participation.  0P983E 
is  responsible  for  across-the-board  JT&E  actions.  Specifically,  0P983E: 

•  Staffs  Navy  JT&E  nominations 

•  Coordinates/reviews  feasibility  studies,  test  designs,  and  test 
plans 

e  Provides  USN  liaison  to  OSD  on  JT&E  policy  issues 
e  Provides  USN  member  of  Planning  Committee 
e  Is  the  initial  USN  contact  for  all  JT&Es 

e  Staffs  the  selection  of  a  sponsor  from  the  CNO  staff  for  each 
test  with  Navy  participation. 

The  sponsor  for  a  particular  JT&E  reviews  the  feasibility  study,  test 
design,  and  test  plan  for  each  joint  test;  staffs  allocation  of  resources 
for  the  test;  and  requests  manpower  billets  and  requisitions  personnel  for 
JTD/DTD  and  the  JTF  staff. 

This  system  provides  a  team  approach  where  the  JT&E  point  of 
contact  is  knowledgeable  concerning  the  entire  spectrum  of  JT&E  activity 
while  the  sponsor(s)  provide(s)  specific  functional  expertise  to  support 
individual  tests. 

b.  Field  Level 

OPTEVFOR,  the  independent  operational  test  agency  for  the 
Navy,  reports  directly  to  the  CNO.  The  mission  of  OPTEVFOR  is  operational 
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Figure  IV- 2. 
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test  and  evaluation  of  specific  weapon  systems,  ships,  aircraft,  and  equip¬ 
ments,  including  procedures  and  tactics  where  required.  When  directed  by 
the  CNO,  OPTEVFOR  assists  development  agencies  in  the  accomplishment  of 
necessary  development  test  and  evaluation.  Up  to  the  present  time, 
OPTEVFOR  has  not  been  involved  extensively  in  Navy  support  of  JT&E.  How¬ 
ever,  OPTEVFOR  is  a  valuable  source  of  expert  testing  advice  or  assistance 
for  a  Navy  JTD/DTD.  Also,  OPTEVFOR  or  its  subordinate  elements  may  provide 
or  administer  player  forces.  OPTEVFOR  has  a  JT&E  point  of  contact  in 
OPTEVFOR  Code  020.  In  brief,  the  Navy  has  a  very  flexible  approach  to  JT&E 
designed  to  treat  each  individual  JT&E  as  a  unique  case.  The  field  element 
which  will  support  the  JTD/OTD  is  selected  to  meet  the  requirements  of  a 
particular  test. 

Normally,  the  CNO  staff  sponsor  will  staff  the  selection  of 
the  field  level  lead  Navy  agency  by  the  time  the  Navy  JTD  or  DTD  is 
approved.  Therefore,  the  JTD,  whether  Navy  or  from  another  Service  is 
provided  the  field  level  structure  by  the  CNO  staff  sponsor. 

One  note  of  caution  is  appropriate.  In  contrast  to  the  Air 
Force  and  Army  where  the  responsible  field  agency  has  definite  responsi¬ 
bilities  and  authorities  related  to  JT&E,  the  Navy  selects  and  delegates 
authorities  and  responsibilties  on  a  case-by-case  basis.  The  JTD  should 
establish  a  clear  understanding  of  his  relationships  and  what  support  will 
be  committed  from  the  field  agency. 

4.  United  States  Air  Force 
a.  Headquarters  Level 

The  Headquarters,  United  States  Air  Force,  point  of  contact 
fur  JT&E  is  in  the  DCS/Operations,  Plans,  and  Readiness  (AF/XO).  The 
abbreviated  Air  Staff  chart  shown  in  Figure  IV-3  depicts  the  JT&E  point  of 
contact  (AF/X00RE)  position  in  the  Air  Staff.  This  office  is  knowledgeable 
on  all  joint  tests.  However,  there  will  be  interested  and  responsible 
persons  elsewhere  on  the  Air  Staff  for  particular  joint  tests.  For 
example,  there  might  be  action  officers  for  a  particular  joint  test  in: 

•  DCS/Research,  Development,  and  Acquisition  (AF/RD) 

•  DCS/logistics  and  Engineering  (AF/LE) 

•  ACS/Studies  and  Analysis  (AF/SA). 


These  action  officers  handle  the  specialized  problems  associated  with  a 
particular  joint  test.  The  AF/XOORE  JT&E  point  of  contact  is  cognizant  of 
those  actions  and  remains  the  single  formal  point  of  contact. 

When  the  JTD  or  OTD  requires  Air  Staff  action  for  personnel 
or  resource  allocations,  the  AF/XOORE  action  officer  is  the  primary 
staffing  agent.  Relationships  among  the  Air  Staff  elements  and  between  the 
Air  Staff  and  elements  in  the  field  are  documented  in  AFR  80-20  "Managing 
the  0S0  Jo’'nt  Test  and  Evaluation  Program." 

In  summary,  for  joint  tests  the  Air  Staff  point  of  contact: 

•  Serves  as  interface  with  OSD  on  JT&E  matters 

•  Publishes  Air  Force  test  directives  on  USAF  participation 

•  Staffs  resource  issues  related  to  joint  tests 

•  Keeps  field  agencies  informed  of  JT&E  developments 

«  Coordinates  personnel  allocations  for  JTD/DTD  and  JTD  staff 

positions 

•  Staffs  USAF  JTD  nominations  and  DTD  appointments  and  forwards 

them  to  OSD 

•  Staffs  JT&E  nominations 

•  Provides  USAF  member  of  JT&E  Planning  Committee 

•  Coordinates  on  JT&E  test  design 

•  Requests  USAF  precedence  rating  in  accordance  with  AFR  27-1 

•  Consolidates  USAF  comments  on  JT&E  test  plans  and  forwards  them 

to  the  JTD. 

b.  Field  Level 
(1)  AFTEC 

Planning  support  and  operational  testing  expertise 
related  to  joint  testing  is  centered  in  the  Air  Force  Test  and  Evaluation 
Center  (AFTEC),  the  USAF  independent  test  and  evaluation  agency  at  Kirtland 
AFB,  New  Mexico.  AFTEC  reports  to  the  Chief  of  Staff  and  has  a  joint 
testing  directorate  which  supports  the  Air  Staff  as  follows: 

•  Leads  USAF  efforts  in  interfacing  with  the  DOTE  Support  Agent 
during  test  feasibility  and  design 
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•  Provides  JT&E  resource  estimates  (documented  in  Joint  Test 
Program  Outl ines--JTP0) 

•  Participates  in  Joint  Test  Director  (JTD)  and  Deputy  Test 
Director  (DTD)  nomination/appointment 

•  Assists  USAF  JTD  in  organizing  his  staff  and  developing  his  test 
plan 

•  Tracks  all  USAF  test  costs 

•  Provides  a  focal  point  for  manpower  and  personnel  support  to  the 
JTF 

•  Administers  OSD  funds  when  USAF  is  Lead  Service 

•  Provides  management  guidance  to  USAF  JTDs/DTDs 

•  Provides  staff  support  to  USAF  DTDs. 

(2)  Major  Commands  (MAJCOMs)  and  Separate  Operating 
Agencies  (SOAs)  — — ^ 

The  MAJCOMs  (e.g. ,  TAC,  SAC)  and  SOAs  provide  most  test 
resources  and  players.  Each  MAJCOM/SOA  maintains  a  permanent  point  of 
contact  for  joint  testing.  JTOs  can  obtain  point  of  contact  information 
from  the  AF/XO  or  AFTEC  JT&E  points  of  contact.  In  general,  the  MAJCOMs/ 
SOAs  assist  joint  testing  by: 

•  Supporting  AFTEC  in  interfacing  with  the  DOTE  Support  Agent 

•  Programming  resources  against  approved  JTPOs 

•  Providing  personnel  and  equipment  as  tasked  in  the  USAF  Test 
Directive 

•  Tracking  test-incurred  costs  (for  provision  to  AFTEC) 

•  Participating  in  nominations  of  JTD/DTD 

•  Evaluating  JT&E  impacts  on  operational  capabilities. 

5.  United  States  Marine  Corps 

a.  Headquarters  Level 

In  the  Marine  Corps,  the  liaison  and  coordination  of  JT&E 
matters  is  performed  by  the  Special  Assistant  for  Operational  Test  and 
Evaluation,  in  the  DC/S  Research,  Development,  and  Studies  (Code  RD  -  See 
Figure  IV-4).  The  staff  element  within  the  Headquarters  which  is  most 
closely  related  to  the  test  functionally  will  be  selected  to  act  as  the 
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Figure  IV-4.  Headquarters  Marine  Corps  Organization 


test  sponsor.  For  example,  the  sponsor  of  a  JT&E  involving  primarily 
aviation  resources  would  probably  be  located  in  the  DC/S  for  Aviation. 

The  Headquarters,  USMC,  Code  RQ  supports  the  JT&E  process  by 
soliciting  USMC  JT&E  nominations  and  providing  USMC  liaison  to  OSD  on 
issues  of  JT&E  policy.  The  USMC  Headquarters  sponsor  selected  for  a  parti¬ 
cular  JT&E  supports  that  JT&E  by  providing  JTD/DTD  nominations  and  recom¬ 
mending  Fleet  Marine  Force  units  to  meet  the  requirements  of  the  JTD  and 
DTD. 

b.  Field  Level 

Significant  authority  has  been  delegated  to  the  Marine  Corps 
Operational  Test  and  Evaluation  Activity  (MCOTEA)  at  Quantico,  Virginia. 
MCOTEA  reports  to  the  Commandant  of  the  Marine  Corps  and  was  chartered  in 
early  1978  as  the  Marine  Corps'  independent  operational  test  and  evaluation 
agency.  Until  now,  USMC  participation  in  joint  testing  has  been  limited. 
Thus,  an  evolution  in  roles,  responsibilities,  and  functions  may  take 
place.  However,  at  present  MCOTEA: 

•  Staffs  the  USMC  position  on  feasibility  studies,  test  designs, 
and  test  plans 

e  Publishes  Test  Planning  Document  for  USMC  participation 

•  Recommends  to  the  Commandant  the  selection  and  tasking  of  the 
Fleet  Marine  Force  for  conduct  or  support  of  JT&E,  if  the  USMC 
Headquarters  sponsor  has  not 

e  Provides  technical  support  to  USMC  JTD/DTD  in  test  planning, 
direction,  and  reporting 

•  Conducts  an  independent  evaluation  if  desired. 

B.  TRAINING 


The  majority  of  personnel  assigned  to  a  Joint  Test  Force  (JTF)  should 
be  fully  qualified  in  their  areas  of  specialization.  Additional  training, 
whether  formal  or  informal,  can  serve  to  update  previously  acquired  skills 
and  can  also  assist  JTF  members  in  transferring  their  expertise  to  the 
joint  testing  environment. 
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1 .  Quadri “Service  Indoctrination  Course 

There  is  currently  only  one  formal  course  which  is  directed 
specifically  toward  joint  test  and  evaluation.  This  course  was  developed 
late  in  1979  by  a  four- service  OT&E  working  committee  staffed  by  represen¬ 
tatives  from  OTEA,  OPTEVFOR,  AFTEC  and  MCQTEA.  It  seeks  to  familiarize  JTF 
members  at  the  middle  and  upper  management  levels  with  the  T&E  organization 
and  methodology  of  each  of  the  four  Services.  The  course  is  offered  twice 
a  year  and  can  be  presented  at  any  location  as  required.  Each  Service 
operational  testing  agency  maintains  a  point  of  contact  for  the  indoc¬ 
trination  course. 

2.  Formal  Training 

The  formal  training  assets  available  within  DoD  may  prove  useful, 
although  the  lengthy  waiting  time  required  may  initially  make  such  courses 
appear  impractical.  If  necessary,  JTF  members  must  be  prepared  to  be 
placed  on  waiting  lists  for  some  courses  in  order  to  take  advantage  of 
cancellations  by  other  students. 

A  comprehensive  list  of  short  (1  day  to  4  weeks)  courses  offered 
within  the  Department  of  Defense  is  contained  in  DoD  5010. 16-C,  the  Defense 
Management  and  Training  (DMET)  catalog.  This  book  lists  the  courses 
offered  along  with  the  twenty  sponsoring  organizations.  Courses  are 
divided  into  broad  categories  ranging  from  automatic  data  processing 
through  financial  management  and  systems  analysis  techniques.  Short 
courses  in  data  management,  contract  administration,  and  procurement  are 
particularly  relevant  to  joint  test  and  evaluation.  Representative  courses 
in  these  areas  contained  in  the  OMET  catalogue  are  listed  in  Table  IV-2. 

3.  Informal  Training 

The  Joint  Test  Oi rector  and  the  Deputy  Test  Director  must  remain 
aware  of  opportunities  to  supplement  formal  training  with  informal  train¬ 
ing.  Many  Service  development  agencies  conduct  informal  training  for 
upgrade  or  refresher  courses  for  their  personnel.  JTDs  or  DTDs  should 
contact  their  respective  development  agencies  for  information  on  training 
available  in  the  particular  area  and  time  of  interest. 
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TABLE  IV-2.  REPRESENTATIVE  SHORT  COURSES  RELATED  TO  JT&E 
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Since  each  JTO  faces  similar  problems,  it  may  prove  useful  to 
send  observers  to  ongoing  JT&Es  to  pick  up  practical  information  on  plan¬ 
ning  and  conducting  tests.  The  management  reports  from  preceding  tests 
also  contain  a  wealth  of  lessons  learned.  Copies  of  these  reports  should 
be  available  from  DOTE. 

C.  JT&E  PROTOCOL 

1 .  Introduction 

Joint  testing  involves  a  sensitive  and  test-specific  tasking 
process.  To  be  effective,  the  JTD  should  understand  the  tasking  process 
and  how  it  can  affect  his  ability  to  meet  test  objectives.  The  basic 
underlying  authority  for  JT&E  and  for  this  manual  is  DoO  Directive  5000.3, 
"Test  and  Evaluation."  Test-specific  authority  for  the  JTD,  the  Lead  Serv¬ 
ice,  and  participating  organizations  is  provided  by  the  DOTE  in  the 
charter  which  establishes  the  Joint  Test  Force.  Memoranda  of  agreement/ 
understanding  are  also  used  to  provide  essential  details  on  tasking, 
reporting  and  support  responsibilities.  These  memoranda  are,  in  general, 
bilateral  agreements  between  the  JTD  and  other  organizations  involved  in 
the  test.  Finally,  this  manual  provides  guidance  for  the  entire  JT&E 
process. 

2.  DoO  Directive  5000.3 

This  Directive  provides  very  general  guidance  on  the  responsi¬ 
bilities,  authorities  and  relationships  of  joint  testing.  In  paragraph 

D. 13.C,  the  Directive  states  that: 

"Control  and  OSD  sponsorship  of  JT&E  will  be  exercised 
by  the  DOTE.  The  DDTE,  in  coordination  with  the  JCS, 
will  task  the  selected  lead  service  or,  through  the 
JCS,  the  selected  CINC  to  conduct  the  test,  incorporate 
the  test  into  joint  exercises  as  appropriate,  appoint  a 
Joint  Test  Director,  develop  the  test  plans,  and  pro¬ 
vide  reports,  as  required." 
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Paragraph  0.12  states  that: 


"Responsibility  for  managing  the  practical  aspects  of 
each  JT&E  will  be  delegated  to  a  specific  DoO  Compo¬ 
nent,  and  supported  by  forces  and  material  from  parti c- 
ipati ng  Components. " 

The  thrust  of  these  two  paragraphs  is  that  the  ODTE  retains  all 
responsibility  and  authority  with  respect  to  test  objectives  and  policy 
while  delegating  to  the  Lead  Service  the  responsibility  for  the  actual 
management  and  conduct  of  the  test.  However,  the  Directive  is  silent  on 
the  authority  which  the  Lead  Service  has  over  "forces  and  material  from 
participating  Components."  Further,  the  Directive  does  not  describe  the 
role,  responsibilities,  and  authorities  delegated  to  the  JTD.  These  two 
important  aspects  of  JT&E  protocol  are  addressed  in  this  manual  and  in  the 
JTD  Charter. 

3.  JTD  Charter 

For  each  test,  the  DOTE  will  prepare  a  Charter  establishing  the 
Joint  Test  Force.  The  .charter  describes  the  mission  of  the  JTD;  designates 
the  Lead  Service  and  participating  Service;  and  provides  tasking  authority, 
authority  for  the  JTD  to  conduct  the  test  within  the  guidelines  of  this 
manual,  and  authority  to  establish  direct  communication  with  supporting 
Servi ces/Agenci es . 

The  Charter  also  specifies  the  channels  through  which  the  JTD 
will  report.  The  JTD  is  usually  directed  to  report  through  the  Lead  Serv¬ 
ice  on  matters  related  to  Service  resources  and  implementation  of  test 
schedules.  For  matters  related  to  DDTE  funding,  test  policy,  test  results, 
and  direction,  the  JTD  reports  directly  to  the  DDTE.  Finally,  the  JTD 
Charter  authorizes  and  encourages  the  JTD  to  communicate  directly  with  all 
organizations  participating  in  or  associated  with  the  test. 

While  the  Charter  contains  specific  tasking  and  grants  general 
authorities,  it  does  not  provide  the  JTD  with  details  on  how  to  create  the 
network  of  relationships  between  the  JTF,  the  DDTE,  and  participating 
Services/agencies.  For  this,  the  JTD  must  rely  on  memoranda  of  under¬ 
standing. 
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k  Memoranda  of  Understanding 

It  is  imperative  that  the  JTD  formally  establish  the  "ground 
rules"  of  relationships  with  all  organizations  with  which  he  must  interact 
during  the  course  of  the  test.  This  is  best  accomplished  by  executing  mem¬ 
oranda  of  understanding,  to  specify  responsibilities,  reporting  channels, 
points  of  contact,  critical  dates,  and  other  important  test-related  infor¬ 
mation.  Failure  to  formally  agree  to  mutual  responsibilities  could  lead  to 
serious  problems  later  in  the  test  (e.g. ,  late  delivery  of  critical 
resources,  insufficient  resources,  etc.).  All  memoranda  of  understanding 
should  be  reviewed  and  revised  as  testing  proceeds  and  requirements  change. 
At  a  minimum,  the  JTD  should  execute  agreements  with: 

•  All  Service  operational  test  agencies 

•  Lead  and  participating  Service  agencies  designated  to  support  the 
test 

•  The  Independent  Evaluation  agency/contractor 

•  Major  potential  users  of  test  results 

•  Ranges/test  sites  to  be  used  during  testing 

•  The  DOTE  Support  Agent  which  prepared  the  Feasibility  Study  and 
Test  Design 

•  Procurement  offices  which  will  be  used  for  contracting  activity. 

5.  Summary 

JT&E  remains  a  challenging  process,  '  demanding  leadership  and 
management  skills  from  all  concerned,  but  most  especially  the  JTD.  The  JTD 
operates  in  a  unique  position.  He  is  responsible  to  the  DDTE  for  conduct 
of  a  JT&E  which  is  invariably  an  event  with  significant  visibility  in  DoD, 
his  Service,  and  the  Congress.  This  manual  has  synthesized  the  experience 
of  present  and  former  JTDs  to  provide  the  fundamentals  of  the  JT&E  process. 
The  manual  contains  enduring  guidance  which  will  apply  in  general  to  all 
JT&Es.  Unique  requirements,  responsibilities,  and  authorities  not  covered 
in  the  manual  should  be  included  in  the  JTD  charter.  To  ensure  its  useful¬ 


ness,  the  manual  will  be  a  dynamic  document  which  is  amended  and  expanded 
as  more  knowledge,  experience,  and  information  becomes  available.  JT&E 
makes  a  significant  contribution  to  improving  and  strengthening  the  Defense 
posture  of  the  United  States.  It  is  imperative  that  all  who  participate 
understand  and  support  the  JT&E  process. 
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APPENDIX  A 

PROGRAM  STATUS  REVIEW 
BRIEFING  CHARTS 


AGENDA 


PROGRAM  STATUS  REVIEW 
DATE 

1.  EVENTS  LISTING 

2.  ACTION  ITEM  STATUS 

3.  TEST  SCHEDULE  STATUS 

4.  CONTRACT(S)  STATUS 

5.  SPECIAL  TOPICS 

6.  JOINT  TEST  DIRECTOR'S  REMARKS 

ooa/sow 


A-1 


M08/900Z 


’ESTIMATED  COMPLETION  DATE 
’’ACTUAL  COMPLETION  DATE 

2  006/8  OW 


DISTRIBUTE  28  ▼ 


M0B/900E 


2006/80W 


AFTEC 

AFXO 

AFXOO 

AFXOOR 

AFXOORE 

CAC 

CDEC 

CINC 

CNO 

COMSEC 

COTR 

CSA 

C/SCS 

OA 

DAMO-RQ 

OAMO-RQT 

OAR 

DCSOPS 

DOTE 

DMET 

OoO 

DTD 

FCRC 

FY 

FYTP 

FYTP 


GLOSSARY  OF  ACRONYMS  USED 
Air  Force  Test  and  Evaluation  Center 

Air  Force  Deputy  Chief  of  Staff  for  Operations,  Plans,  and 
Readiness 

Air  Force  Directorate  for  Operations  and  Readiness 

Air  Force  Deputy  for  Readiness 

Air  Force  Operational  Test  and  Evaluation  Division 

Combined  Arms  Center 

Combat  Developments  Experimentation  Command 
Commander- i n-Chi ef 
Chief  of  Naval  Operations 
Communications  Security 

Contracting  Officer's  Technical  Representative 
Army  Chief  of  Staff 
Cost/Schedule  Control  System 

Department  of  the  Army 

Department  of  the  Army  -  Requirements  Directorate 

Department  of  the  Army  “  Test  Coordination  Office 

Defense  Acquisition  Regulation 

Army  Deputy  Chief  of  Staff  for  Operations  and  Plans 

Director  Defense  Test  and  Evaluation 

Defense  Management  and  Training 

Department  of  Defense 

Oeputy  Test  Director 

Federal  Contract  Research  Center 
Fiscal  Year 

Five-Year  Test  Plan  (DDTE) 

Five-Year  Test  Program  (U.S.  Army) 
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IFFN 

JCS 

JTD 

JT&E 

JTF 

JTPO 

MAJCOM 

MCOTEA 

MIPR 

MOE 

MOU 

ODOTE 

OJCS 

O&M  . 

OMB 

OPSEC 

OPTEVFOR 

OSD 

OT 

0T4E 

OTEA 

OTP 

PE 

POM 

PPBS 


Identification  Friend,  Foe,  or  Neutral 

Joint  Chiefs  of  Staff 
Joint  Test  Oi rector 
Joint  Test  and  Evaluation 
Joint  Test  Force 
Joint  Test  Program  Outline 

Major  Command 

Marine  Corps  Operational  Test  and  Evaluation  Agency 
Military  Interdepartmental  Purchase  Request 
Measure  of  Effectiveness 
Memorandum  of  Understanding 

Office  of  the  Director,  Defense  Test  and  Evaluation 
Office  of  the  Joint  Chiefs  of  Staff 
Operation  and  Maintenance 
Office  of  Management  and  Budget 
Operations  Security 

U.S.  Navy  Operational  Test  and  Evaluation  Force 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Operational  Test  and  Evaluation 

U.S.  Army  Operational  Test  and  Evaluation  Agency 

Outline  Test  Plan 

Program  Element 

Program  Objective  Memoranda 

Planning,  Programming,  and  Budgeting  System 


R&D 

RDT&E 

ROE 


Research  and  Development 

Research,  Development,  Test  and  Evaluation 

Rules  of  Engagement 


SAC 

SAM 

SECDEF 

SOA 

TAB 

TAG 

TCATA 

TECOM 

TEMPEST 

TRADOC 

TSARC 

TSPI 

USALOGC 


Senior  Advisory  Council 
Surface- to-Air  Missile 
Secretary  of  Defense 
Separate  Operating  Agency 

Technical  Advisory  Board 
Technical  Advisory  Group 
TRADOC  Combined  Arms  Test  Activity 
Test  and  Evaluation  Command 
Control  of  Compromising  Emanations 
Training  and  Doctrine  Command 
Test  Schedule  and  Review  Committee 
Time  Space  Position  Information 

U.S.  Army  Logistics  Center 
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INDEX 

Adaptive  Testing,  III-53 

Administration,  11-14 

Analysis  Requirements 
Active  Test,  III-55 
Post-Test,  III-56 
Pre-Test,  III-55 

Analytical  Techniques,  III-6-7,  III-56-59 

Automation,  III-6,  III-16-17,  III-53 

Awards  and  Decorations,  III-65 

Base  Support,  I I 1-21 

Blue  Ribbon  Defense  Panel,  1-2,  III-26 

Budget,  Current  Program,  III-32,  III-33,  III-34,  III-36 

Budget  Estimate,  III-41,  III-42 

Budget  Guidance  Manual,  I I I- 27 

Budgeting,  1-12-13,  III-32,  III-34,  III-36,  III-37,  III-41 

Calibration,  III  S2 

Changes  During  Testing,  III-52-53 

Comparison  Te^nnlques,  III-57 

Computer  Simulation,  III-21-24 

Congress,  1-12 

Contracting,  11-13,  11-16-17 

Contractor  Support,  1-10,  II-5,  11-13,  11-16-17 

Costs,  II 1-26— 29 

Data  Analysis,  III-2,  III-14-16,  III-54-59 
Oata  Collection,  III-14-16,  III-51,  III-53 
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INDEX  (Continued) 


Data  Documentation,  I II- 18 

Data  Elements,  III-6,  III-14,  III-15 

Oata  Management 

During  Active  Testing,  III-53-54 
Plan,  III-14-18 
Process,  III- 15 

Data  Repository,  I I 1-65-66 

Data  Storage  and  Retrieval,  III-66 

Data  Validation,  II 1-51 -52 

Defense  Management  and  Training  Catalog,  IV- 14- 15 

Deputy  Test  Director,  II-8,  11-15 

Director  Defense  Test  and  Evaluation 
Origins,  1-2 

Responsibilities,  II-1-2,  III-5,  IV-17 
Support  Agent,  I- 1 0 ,  1-14,  I 1-5 

Disposition  of  Materiel,  III-62-65 

Documentation,  I I I- 18 

DoD  Budget  Guidance  Manual,  III-27 

DoD  Directive  5000.3,  IV-16-17 

DoD  Planning,  Programming,  and  Budgeting  System,  1-3,  1-12-13 
DoD  Research,  Development,  Test,  and  Evaluation  Program,  1-12 
Dry  Runs,  III-51,  III-55 
Environmental  Protection,  III- 13 
Evaluation  Plan,  III-7-8 

Expenditures  and  Disbursements  Tracking,  I I I - 34 ,  III-46,  III-47 
Feasibility  Determination,  1-11,  1-12,  II-5,  III-4 
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Field  Test  Plan,  1-14,  III-8-14 

Final  Test  Report,  1-15,  III-61 

Financial  Management,  11-17,  II 1-26-48 

Financial  Manager,  11-13,  III-27 

Financial  Planning,  III-26-48 
Essential  Elements,  III-33 

Financial  Tracking  Plan,  III-32,  III-34-35 

Five  Year  Test  Plan,  1-2,  1-3,  1-12 

Five  Year  Test  Program,  IV- 1-3 

Frequency  Allocation,  11-14 

fund  Releases,  III-34,  III-41,  III-43-44 

Funding,  1-12,  1-13,  III-26-29 

Glossary,  Appendix  B 

Graphic  Techniques,  III-58 

Independent  Evaluation  Plan,  1-14,  I 1-5 ,  III-7-8 

Indoctrination  of  JTF,  III-50 

Information  File,  1-15-16 

Instrumentation 

Calibration,  III-52 
Disposition,  III-62-65 
Plan,  I I I- 1 8-20 

Interim  Reports,  1-14,  I 1-3 ,  I 11-59-60 
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Joint  Test  and  Evaluation 
Budgeting,  1-12-13 
Costs,  III-26-29 
Criteria  for  Approval,  1-2 
Cycle,  1-4 

Feasibility  Determination,  I- 1 0 ,  1-12,  I I 1-4 

Financial  Management,  III-26-48 

Funding,  1-12,  III-26-29 

Implementation,  1-13-15,  III-1-66 

Information  File,  1-15-16 

Library,  1-15-16 

Management  Evaluation,  1-15-16 

Management  Report,  1-15,  II-3 

Nomination,  1-3-10 

Planning  Committee,  1-3,  1-7,  1-8 

Planning  Cycle,  1-3,  1-4 

Program  Planning  and  Control,  III-24-28,  Appendix  A 

Protocol,  IV- 16- 18 

Purpose,  1-1 

Scope,  1-1-2 

Selection,  1-3-10 

Senior  Advisory  Council ,  1-7,  1-10,1-11 
Technical  Advisory  Board,  1-7,  1-9 
Test  Report,  1-15,  II-3,  III-61 
Time  Lines,  II-8-9,  III-25 

Joint  Test  Director 
Charter,  IV- 17 

Responsibilities,  1-14-15,  II-2-3,  11-10,  III-l,  III-50,  III-53, 
III— 66 ,  IV- 17,  IV- 18 
Staff  Organization,  II-6-7 

Joint  Test  Force 

Indoctrination,  II 1-50 
Location  Selection,  II-9-10 
Staffing 

PCS,  11-12 
TDY/TAD ,  11-11-12 
Training,  IV-13-16 

JTD  Staff,  I 1-5-1 7 

Mission  and  Functions,  II-5-6 
Organization,  II-6-7 
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Lead  Service,  1-1,  II-3-4,  III-66 
Learning  Effects,  111-54- 

Lessons  Learned,  III-8,  III-17-18,  III-21,  III-25,  III-52,  IV- 1 
Library,  1-15-16 

Logistics  and  Facilities,  II-4,  III-20 

Logistics  Support  Plan,  III-20-21 

Management  Evaluation,  1-15-16 

Management  Report,  1-15,  I 1-3,  II 1-60 

Materiel,  III-20-21 

Materiel  Disposition,  I I 1-62-65 

Memoranda  of  Understanding 
JT&E  Protocol,  IV- 18 
Materiel  Disposition,  III-63 

Nomination,  1-3-10 
Format,  1-6 
Process,  1-5 

Nonparametric  Techniques,  III-58-59 
Obligations-to-Oate,  I 11-37-39 

Office  of  the  Joint  Chiefs  of  Staff  Responsibilities,  1-1,  1-2,  II-2 
Office  of  Management  and  Budget,  1-12 
Pay,  11-15-16 
PCS  Staff,  11-12 

Performance  Evaluations,  11-14-15 
Personnel  Management,  11-11-16,  III-65 
Planning  Committee,  1-3,  1-7,  1-8 
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Post-Test  Activities,  III-3,  III-59-66 

Program  Element  658040,  III-26-27,  III-32,  III-46,  III-48 

Program  Objective  Memoranda,  I- 10 

Program  Planning  and  Control,  11-14,  III-24-48,  Appendix  A 
Program  Status  Review,  III-25-26,  Appendix  A 
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Purpose  of  JT&E,  1-1 

Quadri -Service  Indoctrination  Course,  IV- 14 

Quarterly  Report  of  Obligati  js  and  Billings,  I I 1-48,  I I 1-49 

Quick  Look  Analysis,  III-16,  III-18,  III-54,  III-55 

Rehearsals,  III-51,  III-55 

Reports 

Annual,  1-16 
Interim,  III-59-60 
Management,  1-15,  I I 1-60 
Special,  II 1-60 
Test,  1-15,  I 11-61 

Resour ~e  Manager,  11-13,  III-25,  III-34 

Responsibilities 

DOTE,  II-1-2,  III-5,  IV- 17 
DTD,  II-8 

JTD,  1-14-15,  II-2-3,  11-10,  III-l,  I 11-50 ,  III-53,  III-66 

Lead  Service,  1-1,  II-3-4,  III-28-29,  III-66 

OJCS,  1-1,  1-2,  1-5,  I 1-2 

Planning  Committee,  1-3,  1-7,  1-8 

Senior  Advisory  Council,  1-7,  I- 1 0 ,  1-11 

Services,  II-4-5,  III-28-29 

Support  Agent,  II-5 

Technical  Advisory  Board,  1-7,  1-9 

Risks,  I 11-48 

Senior  Advisory  Council,  1-7,  1-10,  1-11 
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Schedule,  II-8-9,  III-25-26 
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Security,  11-13,  III-13 
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Advantages,  III-22-23 
Disadvantages,  I 11-23 
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Software  Development,  III-16 
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Spending  Plan,  I 11-39-40 
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Support  Agent,  1-10,  1-14,  II-5 

TDY/TAD  Staff,  11-11-12 
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